SPB Git
spboucher.ai

spb/wp31_uqo

Public

WP31: 5-seed routing summary, statistics, ranking controller result

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Oct 30, 2026) parent 512b980 Browse files

26 changed files +1,850 −34

26 changed files
Unified Split
modified FAILURES.md +1 −0
@@ -10,3 +10,4 @@
10 | 2026-10-30 | Joint fine-tuning reference (backbone unfrozen, lr 3e-4, answer-span loss): validation loss rose from 0.4005 to 0.472 at every budget after 500 steps — the backbone degraded under the narrower objective and the large learning rate. | M4M64a `experiments/logs/p4_joint_mixed_w64.log` | Not a usable upper reference as run; a low-lr variant is deferred (not required by the pre-registered comparison). | 10 | 2026-10-30 | Joint fine-tuning reference (backbone unfrozen, lr 3e-4, answer-span loss): validation loss rose from 0.4005 to 0.472 at every budget after 500 steps — the backbone degraded under the narrower objective and the large learning rate. | M4M64a `experiments/logs/p4_joint_mixed_w64.log` | Not a usable upper reference as run; a low-lr variant is deferred (not required by the pre-registered comparison). |
11 | 2026-10-30 | v2 tiny backbone (5.6M, 15k planned steps) overfit the 23k-item train split: validation loss (all tokens) 0.246 at step 6k → 0.282 at step 9k. Run stopped at step 9,300; the step-5,000 checkpoint (val 0.249) is used as the frozen v2 backbone. | `experiments/logs/p4_tiny_backbone_v2.log`, run `p4_tiny_backbone_v2_seed01_*` (state: stopped) | 42 epochs of a small synthetic set is too many; schedule not re-run (the checkpoint is adequate as a frozen comparator). | 11 | 2026-10-30 | v2 tiny backbone (5.6M, 15k planned steps) overfit the 23k-item train split: validation loss (all tokens) 0.246 at step 6k → 0.282 at step 9k. Run stopped at step 9,300; the step-5,000 checkpoint (val 0.249) is used as the frozen v2 backbone. | `experiments/logs/p4_tiny_backbone_v2.log`, run `p4_tiny_backbone_v2_seed01_*` (state: stopped) | 42 epochs of a small synthetic set is too many; schedule not re-run (the checkpoint is adequate as a frozen comparator). |
12 | 2026-10-30 | Killed to free nodes for the 5-seed matrix: the degraded joint-finetune reference (M4M64a) and the v1 dense-writes run on the slow m4mn node (superseded by the v2 matrix `p4v2_dense`). | manifests remain in `experiments/runs/` on the nodes (state: running → killed) | — | 12 | 2026-10-30 | Killed to free nodes for the 5-seed matrix: the degraded joint-finetune reference (M4M64a) and the v1 dense-writes run on the slow m4mn node (superseded by the v2 matrix `p4v2_dense`). | manifests remain in `experiments/runs/` on the nodes (state: running → killed) | — |
13+| 2026-10-30 | Ranking-loss controller (D12: pairwise + listwise on top of Huber/BCE/NLL, editor-unseen labels): calibration sign accuracy fell to 45 %, captured fraction −8 %; on dev-iid the locked policy (λ=0.3, κ=0.5) edits 0.07 tokens and matches budget 0 (0.4333 vs 0.4334); λ=0 is worse (+1.3 %). | `experiments/runs/controller_v1rank_seed01_*`, `results/raw/routing_v1rank.json` | Within-state ranking is not learnable from these labels at this scale; H3 null stands. |
modified PROGRESS.md +3 −1
@@ -9,9 +9,11 @@
9 | 2026-10-29 | 4 learnability (backbone) | done | `p4_tiny_backbone.yaml` → run `p4_tiny_backbone_seed01_20261029232626` (2.1M params, 6k steps, char-level) | exact match (greedy): dev_iid arith 16.8 %, dev_template 13.7 %, dev_depth 3.0 %, compose 2.4 % (first-600-item subsets, arith-heavy) | PASS (weak backbone, usable comparator) | stronger v2 backbone (5.6M, 15k steps) running | 9 | 2026-10-29 | 4 learnability (backbone) | done | `p4_tiny_backbone.yaml` → run `p4_tiny_backbone_seed01_20261029232626` (2.1M params, 6k steps, char-level) | exact match (greedy): dev_iid arith 16.8 %, dev_template 13.7 %, dev_depth 3.0 %, compose 2.4 % (first-600-item subsets, arith-heavy) | PASS (weak backbone, usable comparator) | stronger v2 backbone (5.6M, 15k steps) running |
10 | 2026-10-29 | 4 learnability (overfit) | done | `p4_overfit64_b8.yaml` → run `p4_overfit64_b8_seed01_20261029235111` | 64 items, budget 8, frozen backbone: train loss 0.347 → 0.0024 in 200 steps; dev loss rises 0.73 → 1.36 (expected) | PASS (gradient path + capacity) | — | 10 | 2026-10-29 | 4 learnability (overfit) | done | `p4_overfit64_b8.yaml` → run `p4_overfit64_b8_seed01_20261029235111` | 64 items, budget 8, frozen backbone: train loss 0.347 → 0.0024 in 200 steps; dev loss rises 0.73 → 1.36 (expected) | PASS (gradient path + capacity) | — |
11 | 2026-10-29 | 4 learnability (generalisation, v1 backbone) | done | zero-U scaled (laptop), U std 0.2 / 0.5 (M4M64c / M2U64), budget-8-only w64/w128 and mixed w128 lr 3e-3 (M2U64 / M4M64c / M4M64a) | after 2000 steps (U std 0.2): dev-iid loss −1.4 % at budgets 1–2, +1.7 % at budget 8; exact match dev-iid 43.8 → 45.8 % (b2), template/depth unchanged, compose 2.3 → 3.0 %; zero-U + scale alone: −0.6 %; lr 3e-3 w128: collapse to inert edits; global-recurrence adapter (same recipe): loss ≥ backbone at every budget, EM 43.8 → 45.8 % (b8) | PASS (weak, learnable; mechanism effect ≈ 1 %) | decisive 5-seed matrix on v2 backbone | 11 | 2026-10-29 | 4 learnability (generalisation, v1 backbone) | done | zero-U scaled (laptop), U std 0.2 / 0.5 (M4M64c / M2U64), budget-8-only w64/w128 and mixed w128 lr 3e-3 (M2U64 / M4M64c / M4M64a) | after 2000 steps (U std 0.2): dev-iid loss −1.4 % at budgets 1–2, +1.7 % at budget 8; exact match dev-iid 43.8 → 45.8 % (b2), template/depth unchanged, compose 2.3 → 3.0 %; zero-U + scale alone: −0.6 %; lr 3e-3 w128: collapse to inert edits; global-recurrence adapter (same recipe): loss ≥ backbone at every budget, EM 43.8 → 45.8 % (b8) | PASS (weak, learnable; mechanism effect ≈ 1 %) | decisive 5-seed matrix on v2 backbone |
12−| 2026-10-30 | 4 decisive matrix (v2 backbone, 5 seeds × {PRISME, global recurrence, dense}) | running | `scripts/launch_seed_matrix.sh` → 15 runs on M2U64/M4M64c/M4M64a/M5M36/M4BP48/M4BP36 | — | — | aggregate with `scripts/make_tables.py` | 12+| 2026-10-30 | 4 decisive matrix (v2 backbone, 5 seeds × {PRISME, global recurrence}; dense running) | done (PRISME, GR) | `scripts/launch_seed_matrix.sh`; `scripts/statistics.py` → `results/analyzed/{seed_matrix_v2,statistics_v2}.json` | dev-iid loss vs budget 0 (0.4621): PRISME −2.1/−2.8/−3.0/−1.0 % at b=1/2/4/8 (seed SD ≤ 0.003); GR −3.6/−4.1/−3.7/−3.3 %; EM dev-iid 37.7 → PRISME 39.6 (b2, +1.9 pp, t-CI [1.6, 2.2]) vs GR 39.6 (b2) / 40.0 (b8); PRISME − GR at b2 +0.06 pp [−0.53, +0.65]; OOD regimes unchanged (≤ 1 pp) | H1 weak PASS (< 3 pp prereg minimum); sparse ≤ global at every budget | dense seeds pending (M4BP36 lost; relaunched) |
13+| 2026-10-30 | 6 controller on v2 editors (4 seeds) | done | `scripts/run_stage_cd.sh` → `results/analyzed/routing_seeds_v2.json` | locked controller: dev-iid loss +0.5 % ± 1.1 vs budget 0 (0.05–4.3 edits/token); best fixed schedule −4.0 %; oracle b8 −15.0 %; calib sign acc 0.50–0.63, captured 0.30–0.52 on calib audit but no transfer to dev rollouts | H3 null replicated across seeds | ranking-loss variant (D12) |
13 | 2026-10-30 | 5 counterfactual labels + oracle audit (tiny) | done | `labels_p4.yaml` → run `labels_p4_seed01_20261030000543` (80,632 train + 20,040 calib labels; 4,778 + 1,308 exhaustive-audit states); `evaluation.label_analysis` | gains: mean 0.004, 42 % positive, 37 % harmful; variance 86–89 % between states vs 11–14 % within state (action identity); 37 % of states have all 24 actions positive, 29 % all negative; one-step oracle 0.0166 nats/state = 0.0098 (perfect stop, random action) + 0.0067 (action selection); random action 0.0004 | PASS (oracle margin exists; action choice is ≈40 % of it) | controller fit | 14 | 2026-10-30 | 5 counterfactual labels + oracle audit (tiny) | done | `labels_p4.yaml` → run `labels_p4_seed01_20261030000543` (80,632 train + 20,040 calib labels; 4,778 + 1,308 exhaustive-audit states); `evaluation.label_analysis` | gains: mean 0.004, 42 % positive, 37 % harmful; variance 86–89 % between states vs 11–14 % within state (action identity); 37 % of states have all 24 actions positive, 29 % all negative; one-step oracle 0.0166 nats/state = 0.0098 (perfect stop, random action) + 0.0067 (action selection); random action 0.0004 | PASS (oracle margin exists; action choice is ≈40 % of it) | controller fit |
14 | 2026-10-30 | 6 controller (tiny) | partial | `controller_p4.yaml` → `controller_p4_seed01_20261030000716` (z features): calib sign acc 49 %, captured fraction −3 %; sweep: best (λ=0.1, κ=0.5) loss −0.1 % with 0.01 edits/token; `controller_p4_hent.yaml` (h + backbone entropy): sign acc 59–60 % at 1–2k steps then overfits (early stopping added) | controller with z-only features = chance; h+entropy features predict the sign above chance but capture ≤ 5 % of the oracle opportunity | OPEN (H3 null so far at tiny scale) | rerun with early stopping; more labels | 15 | 2026-10-30 | 6 controller (tiny) | partial | `controller_p4.yaml` → `controller_p4_seed01_20261030000716` (z features): calib sign acc 49 %, captured fraction −3 %; sweep: best (λ=0.1, κ=0.5) loss −0.1 % with 0.01 edits/token; `controller_p4_hent.yaml` (h + backbone entropy): sign acc 59–60 % at 1–2k steps then overfits (early stopping added) | controller with z-only features = chance; h+entropy features predict the sign above chance but capture ≤ 5 % of the oracle opportunity | OPEN (H3 null so far at tiny scale) | rerun with early stopping; more labels |
15 | 2026-10-30 | 7 inference engine + latency (tiny) | done | `tests/integration/test_cache_equivalence.py` (5 tests pass); `evaluation.latency` on the laptop MPS → `results/raw/latency_p4_*_mps.json` | batch-1 decode: 2.8 ms/token (budget 0) → 5.3 (1 edit) → 19.3 ms (8 edits); dense 8 edits 17.3 ms (not slower than sparse at this scale); controller call ≈ +2.9 ms even with 0 edits | PASS (equivalence); dispatch overhead dominates at tiny scale (brief §13.6 confirmed) | re-measure at 103M | 16 | 2026-10-30 | 7 inference engine + latency (tiny) | done | `tests/integration/test_cache_equivalence.py` (5 tests pass); `evaluation.latency` on the laptop MPS → `results/raw/latency_p4_*_mps.json` | batch-1 decode: 2.8 ms/token (budget 0) → 5.3 (1 edit) → 19.3 ms (8 edits); dense 8 edits 17.3 ms (not slower than sparse at this scale); controller call ≈ +2.9 ms even with 0 edits | PASS (equivalence); dispatch overhead dominates at tiny scale (brief §13.6 confirmed) | re-measure at 103M |
16 | 2026-10-29 | 8 pilot (Stage A) | running | M3U96a: `pretrain_103m.yaml` (run `pretrain103m_seed01_*`); M3U96b: `pretrain_enlarged_102m.yaml` | 14.8k tok/s, 196.6M tokens planned (~3.7 h each) | — | editor/controller stages after completion | 17 | 2026-10-29 | 8 pilot (Stage A) | running | M3U96a: `pretrain_103m.yaml` (run `pretrain103m_seed01_*`); M3U96b: `pretrain_enlarged_102m.yaml` | 14.8k tok/s, 196.6M tokens planned (~3.7 h each) | — | editor/controller stages after completion |
17 | 2026-10-30 | 5/6 routing evaluation (tiny v1 editor, dev-iid, 640 items / 3,417 answer tokens) | done | `evaluation.routing` → `results/raw/routing_controller_p4_hent_seed01_20261030002015_dev_iid.json` | budget 0: 0.4334; uniform b1 0.4301 (−0.8 %), random b1 0.4305, fixed pair b1 0.4306; b8 schedules +0.6 to +2.8 %; entropy router ≥ budget 0; norm router (thr 0.2) 0.4309; learned controller (locked λ=0.3) 0.4404 (+1.6 %) and worse at every λ; greedy one-step oracle 0.4025 (b1) → 0.3666 (b8, 2.26 edits/token, −15 %); two-step oracle gain 0.034 vs one-step 0.0233 vs greedy-then-best 0.0337 (myopia regret ≈ 0) | H2 null (uniform = random = pair); H3 null (controller < no routing on dev) at this scale; oracle opportunity large | labels from a split unseen by the editor (D11); 5-seed v2 matrix | 18 | 2026-10-30 | 5/6 routing evaluation (tiny v1 editor, dev-iid, 640 items / 3,417 answer tokens) | done | `evaluation.routing` → `results/raw/routing_controller_p4_hent_seed01_20261030002015_dev_iid.json` | budget 0: 0.4334; uniform b1 0.4301 (−0.8 %), random b1 0.4305, fixed pair b1 0.4306; b8 schedules +0.6 to +2.8 %; entropy router ≥ budget 0; norm router (thr 0.2) 0.4309; learned controller (locked λ=0.3) 0.4404 (+1.6 %) and worse at every λ; greedy one-step oracle 0.4025 (b1) → 0.3666 (b8, 2.26 edits/token, −15 %); two-step oracle gain 0.034 vs one-step 0.0233 vs greedy-then-best 0.0337 (myopia regret ≈ 0) | H2 null (uniform = random = pair); H3 null (controller < no routing on dev) at this scale; oracle opportunity large | labels from a split unseen by the editor (D11); 5-seed v2 matrix |
19+| 2026-10-30 | 6 ranking-loss controller (D12, tiny v1 editor, editor-unseen labels) | done | `controller_v1rank.yaml` → `results/raw/routing_v1rank.json` | locked λ=0.3 κ=0.5: dev-iid loss 0.4333 vs 0.4334 (0.07 edits/token); λ=0: +1.3 % | H3 null stands | — |
modified configs/experiments/routing_v1rank.yaml +1 −1
@@ -1,7 +1,7 @@
1 seed: 1 1 seed: 1
2 mode: prisme 2 mode: prisme
3 model: from_checkpoint 3 model: from_checkpoint
4−checkpoint: CONTROLLER_CKPT 4+checkpoint: experiments/runs/controller_v1rank_seed01_20261030005431/checkpoint
5 data: 5 data:
6 kind: synthetic 6 kind: synthetic
7 tokenizer: char 7 tokenizer: char
modified experiments/logs/controller_v1rank.log +10 −0
@@ -33,3 +33,13 @@ step 1999 loss 3.74421 gain 0.00241 harm 0.5247
33 restored controller from step 750 (calibration score -0.01242) 33 restored controller from step 750 (calibration score -0.01242)
34 sweep {'lambda': 0.0, 'kappa': 0.0, 'loss': 0.44029377250626134, 'edits_per_token': 3.0255041518386716, 'extra_ops_ratio': 0.9985018783334029} 34 sweep {'lambda': 0.0, 'kappa': 0.0, 'loss': 0.44029377250626134, 'edits_per_token': 3.0255041518386716, 'extra_ops_ratio': 0.9985018783334029}
35 sweep {'lambda': 0.0, 'kappa': 0.5, 'loss': 0.43276796307003795, 'edits_per_token': 1.5634638196915778, 'extra_ops_ratio': 0.5107896924725914} 35 sweep {'lambda': 0.0, 'kappa': 0.5, 'loss': 0.43276796307003795, 'edits_per_token': 1.5634638196915778, 'extra_ops_ratio': 0.5107896924725914}
36+sweep {'lambda': 0.01, 'kappa': 0.0, 'loss': 0.4399834968865554, 'edits_per_token': 2.7941874258600237, 'extra_ops_ratio': 0.9065133996422746}
37+sweep {'lambda': 0.01, 'kappa': 0.5, 'loss': 0.4311416666702868, 'edits_per_token': 1.3849347568208779, 'extra_ops_ratio': 0.44641059432024066}
38+sweep {'lambda': 0.03, 'kappa': 0.0, 'loss': 0.4382320236622368, 'edits_per_token': 2.3718861209964412, 'extra_ops_ratio': 0.7466290384566374}
39+sweep {'lambda': 0.03, 'kappa': 0.5, 'loss': 0.43146511272462096, 'edits_per_token': 1.120403321470937, 'extra_ops_ratio': 0.3504910203202905}
40+sweep {'lambda': 0.1, 'kappa': 0.0, 'loss': 0.43422956681619196, 'edits_per_token': 1.2627520759193358, 'extra_ops_ratio': 0.371382664672562}
41+sweep {'lambda': 0.1, 'kappa': 0.5, 'loss': 0.4304978969116935, 'edits_per_token': 0.5195729537366548, 'extra_ops_ratio': 0.15240548768501688}
42+sweep {'lambda': 0.3, 'kappa': 0.0, 'loss': 0.4298561159543437, 'edits_per_token': 0.3309608540925267, 'extra_ops_ratio': 0.09634915280596641}
43+sweep {'lambda': 0.3, 'kappa': 0.5, 'loss': 0.4300144485165923, 'edits_per_token': 0.11684460260972716, 'extra_ops_ratio': 0.034015740434752945}
44+selected {'lambda': 0.3, 'kappa': 0.5, 'loss': 0.4300144485165923, 'edits_per_token': 0.11684460260972716, 'extra_ops_ratio': 0.034015740434752945, 'loss_reduction': 0.001048727250466841, 'reduction_per_ops': 0.03083064596163796}
45+run controller_v1rank_seed01_20261030005431
added experiments/logs/routing_v1rank.log +36 −0
@@ -0,0 +1,36 @@
1+{"policy": "budget0", "loss": 0.43343788690653534, "edits_per_token": 0.0, "extra_ops_ratio": 0.0, "n_tokens": 3417}
2+/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/training/train_editor.py:57: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /Users/runner/work/pytorch/pytorch/torch/csrc/utils/tensor_numpy.cpp:214.)
3+ return torch.from_numpy(acts).to(device), torch.from_numpy(np.ascontiguousarray(b)).to(device)
4+{"policy": "random_forced", "budget": 1, "loss": 0.43049107206331766, "edits_per_token": 1.0, "extra_ops_ratio": 0.33739269034430613, "n_tokens": 3417}
5+{"policy": "uniform", "budget": 1, "loss": 0.430082754325476, "edits_per_token": 1.0, "extra_ops_ratio": 0.2911194881431394, "n_tokens": 3417}
6+{"policy": "fixed_pair", "budget": 1, "loss": 0.4306084942950259, "edits_per_token": 1.0, "extra_ops_ratio": 0.38326141294904154, "n_tokens": 3417}
7+{"policy": "random_forced", "budget": 2, "loss": 0.43129151572879726, "edits_per_token": 2.0, "extra_ops_ratio": 0.6752437784687831, "n_tokens": 3417}
8+{"policy": "uniform", "budget": 2, "loss": 0.43061076446970825, "edits_per_token": 2.0, "extra_ops_ratio": 0.5822389762862789, "n_tokens": 3417}
9+{"policy": "fixed_pair", "budget": 2, "loss": 0.43226337237493456, "edits_per_token": 2.0, "extra_ops_ratio": 0.7665228258980831, "n_tokens": 3417}
10+{"policy": "random_forced", "budget": 4, "loss": 0.4350665061488499, "edits_per_token": 4.0, "extra_ops_ratio": 1.3500830851558498, "n_tokens": 3417}
11+{"policy": "uniform", "budget": 4, "loss": 0.43226772879940184, "edits_per_token": 4.0, "extra_ops_ratio": 1.1644779525725577, "n_tokens": 3417}
12+{"policy": "fixed_pair", "budget": 4, "loss": 0.43608775067685257, "edits_per_token": 4.0, "extra_ops_ratio": 1.5330456517961661, "n_tokens": 3417}
13+{"policy": "random_forced", "budget": 8, "loss": 0.44534724575753865, "edits_per_token": 8.0, "extra_ops_ratio": 2.6992224028210283, "n_tokens": 3417}
14+{"policy": "uniform", "budget": 8, "loss": 0.4360636224654602, "edits_per_token": 8.0, "extra_ops_ratio": 2.3289559051451154, "n_tokens": 3417}
15+{"policy": "fixed_pair", "budget": 8, "loss": 0.4454028259636821, "edits_per_token": 8.0, "extra_ops_ratio": 3.0660913035923323, "n_tokens": 3417}
16+{"policy": "entropy", "threshold": 0.1, "budget": 8, "loss": 0.44345039856653656, "edits_per_token": 2.893473807433421, "extra_ops_ratio": 0.9313065950585138, "n_tokens": 3417}
17+{"policy": "entropy", "threshold": 0.5, "budget": 8, "loss": 0.44244280132174246, "edits_per_token": 2.097453906935909, "extra_ops_ratio": 0.6750847836283169, "n_tokens": 3417}
18+{"policy": "entropy", "threshold": 1.0, "budget": 8, "loss": 0.4375149400603328, "edits_per_token": 1.1785191688615744, "extra_ops_ratio": 0.3794397201180842, "n_tokens": 3417}
19+{"policy": "entropy", "threshold": 2.0, "budget": 8, "loss": 0.43457157094819804, "edits_per_token": 0.20749195200468246, "extra_ops_ratio": 0.06636437945523596, "n_tokens": 3417}
20+{"policy": "norm", "threshold": 0.02, "budget": 8, "loss": 0.4431566372928056, "edits_per_token": 7.994146912496342, "extra_ops_ratio": 2.576118717076383, "n_tokens": 3417}
21+{"policy": "norm", "threshold": 0.05, "budget": 8, "loss": 0.4419520246658124, "edits_per_token": 7.054141059408838, "extra_ops_ratio": 2.2736114108984604, "n_tokens": 3417}
22+{"policy": "norm", "threshold": 0.1, "budget": 8, "loss": 0.4397261778374042, "edits_per_token": 4.717588527948493, "extra_ops_ratio": 1.5199677263884506, "n_tokens": 3417}
23+{"policy": "norm", "threshold": 0.2, "budget": 8, "loss": 0.430901125784258, "edits_per_token": 1.0491659350307287, "extra_ops_ratio": 0.33984092665514054, "n_tokens": 3417}
24+{"policy": "controller_locked", "lam": 0.3, "kap": 0.5, "budget": 8, "loss": 0.43332059026845965, "edits_per_token": 0.06613988879133743, "extra_ops_ratio": 0.01925461050499103, "n_tokens": 3417}
25+{"policy": "controller", "lam": 0.0, "kap": 0.5, "budget": 8, "loss": 0.4392198074760582, "edits_per_token": 1.4641498390400935, "extra_ops_ratio": 0.48109084149707576, "n_tokens": 3417}
26+{"policy": "controller", "lam": 0.01, "kap": 0.5, "budget": 8, "loss": 0.43936958569757484, "edits_per_token": 1.3075797483172373, "extra_ops_ratio": 0.42456216117182355, "n_tokens": 3417}
27+{"policy": "controller", "lam": 0.03, "kap": 0.5, "budget": 8, "loss": 0.438084998115448, "edits_per_token": 1.0398009950248757, "extra_ops_ratio": 0.3312630400798598, "n_tokens": 3417}
28+{"policy": "controller", "lam": 0.1, "kap": 0.5, "budget": 8, "loss": 0.4354970293991183, "edits_per_token": 0.4433713784021071, "extra_ops_ratio": 0.13166276062654758, "n_tokens": 3417}
29+{"policy": "controller", "lam": 0.3, "kap": 0.5, "budget": 8, "loss": 0.43332059026845965, "edits_per_token": 0.06613988879133743, "extra_ops_ratio": 0.01925461050499103, "n_tokens": 3417}
30+{"policy": "oracle1_greedy", "budget": 1, "diagnostic": true, "loss": 0.4025459464804867, "edits_per_token": 0.5237366003062787, "extra_ops_ratio": 0.19226168855956283, "n_tokens": 653}
31+{"policy": "oracle1_greedy", "budget": 2, "diagnostic": true, "loss": 0.390496985653093, "edits_per_token": 0.9509954058192955, "extra_ops_ratio": 0.34655941646277993, "n_tokens": 653}
32+{"policy": "oracle1_greedy", "budget": 4, "diagnostic": true, "loss": 0.37995706181431255, "edits_per_token": 1.5313935681470139, "extra_ops_ratio": 0.552776494398599, "n_tokens": 653}
33+{"policy": "oracle1_greedy", "budget": 8, "diagnostic": true, "loss": 0.3665898426751083, "edits_per_token": 2.2664624808575806, "extra_ops_ratio": 0.8196839220125861, "n_tokens": 653}
34+{"policy": "oracle_two_step", "diagnostic": true, "one_step_gain": 0.023339285247627347, "two_step_gain": 0.03395615775009681, "greedy_then_best_gain": 0.03374234835306803, "n_tokens": 174}
35+wrote results/raw/routing_v1rank.json
36+RANK_DONE
added experiments/runs/controller_v1rank_seed01_20261030005431/checkpoint/checkpoint.json +49 −0
@@ -0,0 +1,49 @@
1+{
2+ "config": {
3+ "name": "tiny_p4",
4+ "vocab_size": 128,
5+ "d_model": 192,
6+ "n_layers": 4,
7+ "n_heads": 6,
8+ "n_kv_heads": 2,
9+ "ffn_width": 512,
10+ "max_seq_len": 320,
11+ "rope_theta": 10000.0,
12+ "norm_eps": 1e-05,
13+ "tie_embeddings": true,
14+ "init_std": 0.02,
15+ "use_workspace": true,
16+ "w": 64,
17+ "ffn_w": 192,
18+ "n_slots": 16,
19+ "n_editor_blocks": 2,
20+ "n_editor_heads": 2,
21+ "r_max": 8,
22+ "eta_max": 0.25,
23+ "clip_c": 1.0,
24+ "clip_eps": 1e-06,
25+ "readout_zero_init": false,
26+ "readout_init_std": 0.2,
27+ "readout_scale": 5.251033782958984,
28+ "controller_hidden": 0,
29+ "ctrl_input": "h",
30+ "ctrl_use_entropy": true,
31+ "lambda_cost": 0.3,
32+ "kappa_risk": 0.5,
33+ "dense_writes": false
34+ },
35+ "saved_at": "2026-10-30T00:57:23-0400",
36+ "run_id": "controller_v1rank_seed01_20261030005431",
37+ "parent": "experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/checkpoint",
38+ "selected": {
39+ "lambda": 0.3,
40+ "kappa": 0.5,
41+ "loss": 0.4300144485165923,
42+ "edits_per_token": 0.11684460260972716,
43+ "extra_ops_ratio": 0.034015740434752945,
44+ "loss_reduction": 0.001048727250466841,
45+ "reduction_per_ops": 0.03083064596163796
46+ },
47+ "locked": true,
48+ "weights_sha256": "64570e628212ba370db86265ffaa23f655bfee6e58120829c5bbe5881c7aadd9"
49+}
\ No newline at end of file
modified experiments/runs/controller_v1rank_seed01_20261030005431/manifest.json +136 −3
@@ -9,10 +9,143 @@
9 "python": "3.12.13", 9 "python": "3.12.13",
10 "torch": "2.15.0", 10 "torch": "2.15.0",
11 "command": "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/training/train_controller.py --config configs/experiments/controller_v1rank.yaml --device mps", 11 "command": "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/training/train_controller.py --config configs/experiments/controller_v1rank.yaml --device mps",
12− "state": "running", 12+ "state": "completed",
13− "wall_seconds": 0.02, 13+ "wall_seconds": 171.86,
14 "device": "mps", 14 "device": "mps",
15 "checkpoint_parent": "experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/checkpoint", 15 "checkpoint_parent": "experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/checkpoint",
16 "labels_train": "experiments/runs/labels_v1ctrl_seed01_20261030004018/labels_train.pt", 16 "labels_train": "experiments/runs/labels_v1ctrl_seed01_20261030004018/labels_train.pt",
17− "labels_calib": "experiments/runs/labels_v1ctrl_seed01_20261030004018/labels_calib.pt" 17+ "labels_calib": "experiments/runs/labels_v1ctrl_seed01_20261030004018/labels_calib.pt",
18+ "finished_at": "2026-10-30T00:57:23-0400",
19+ "eval_train": {
20+ "huber": 0.0022109991413126954,
21+ "sign_acc": 0.5578782919208451,
22+ "harm_brier": 0.18087974557421105,
23+ "harm_ece": 0.05984468385577202,
24+ "n_labels": 26884,
25+ "oracle_opportunity_per_state": 0.015146345818821804,
26+ "regret_per_state": 0.012184017977496993,
27+ "captured_fraction": 0.19558036484573305,
28+ "top1_match": 0.05240443896424168,
29+ "n_audit_states": 1622
30+ },
31+ "eval_calib": {
32+ "huber": 0.002075914168963728,
33+ "sign_acc": 0.5701813048001189,
34+ "harm_brier": 0.18367661889833475,
35+ "harm_ece": 0.06215643882751465,
36+ "n_labels": 26916,
37+ "oracle_opportunity_per_state": 0.013603728955277556,
38+ "regret_per_state": 0.012418674625524272,
39+ "captured_fraction": 0.08711246259383487,
40+ "top1_match": 0.06930091185410335,
41+ "n_audit_states": 1645
42+ },
43+ "sweep": [
44+ {
45+ "lambda": 0.0,
46+ "kappa": 0.0,
47+ "loss": 0.44029377250626134,
48+ "edits_per_token": 3.0255041518386716,
49+ "extra_ops_ratio": 0.9985018783334029,
50+ "loss_reduction": -0.00923059673920218,
51+ "reduction_per_ops": -0.00924444604411656
52+ },
53+ {
54+ "lambda": 0.0,
55+ "kappa": 0.5,
56+ "loss": 0.43276796307003795,
57+ "edits_per_token": 1.5634638196915778,
58+ "extra_ops_ratio": 0.5107896924725914,
59+ "loss_reduction": -0.001704787302978794,
60+ "reduction_per_ops": -0.003337552280521541
61+ },
62+ {
63+ "lambda": 0.01,
64+ "kappa": 0.0,
65+ "loss": 0.4399834968865554,
66+ "edits_per_token": 2.7941874258600237,
67+ "extra_ops_ratio": 0.9065133996422746,
68+ "loss_reduction": -0.008920321119496255,
69+ "reduction_per_ops": -0.009840252910785835
70+ },
71+ {
72+ "lambda": 0.01,
73+ "kappa": 0.5,
74+ "loss": 0.4311416666702868,
75+ "edits_per_token": 1.3849347568208779,
76+ "extra_ops_ratio": 0.44641059432024066,
77+ "loss_reduction": -7.849090322764596e-05,
78+ "reduction_per_ops": -0.00017582670354668846
79+ },
80+ {
81+ "lambda": 0.03,
82+ "kappa": 0.0,
83+ "loss": 0.4382320236622368,
84+ "edits_per_token": 2.3718861209964412,
85+ "extra_ops_ratio": 0.7466290384566374,
86+ "loss_reduction": -0.0071688478951776324,
87+ "reduction_per_ops": -0.009601619446782316
88+ },
89+ {
90+ "lambda": 0.03,
91+ "kappa": 0.5,
92+ "loss": 0.43146511272462096,
93+ "edits_per_token": 1.120403321470937,
94+ "extra_ops_ratio": 0.3504910203202905,
95+ "loss_reduction": -0.00040193695756179926,
96+ "reduction_per_ops": -0.0011467824687619549
97+ },
98+ {
99+ "lambda": 0.1,
100+ "kappa": 0.0,
101+ "loss": 0.43422956681619196,
102+ "edits_per_token": 1.2627520759193358,
103+ "extra_ops_ratio": 0.371382664672562,
104+ "loss_reduction": -0.0031663910491327996,
105+ "reduction_per_ops": -0.008525952744521666
106+ },
107+ {
108+ "lambda": 0.1,
109+ "kappa": 0.5,
110+ "loss": 0.4304978969116935,
111+ "edits_per_token": 0.5195729537366548,
112+ "extra_ops_ratio": 0.15240548768501688,
113+ "loss_reduction": 0.0005652788553656696,
114+ "reduction_per_ops": 0.0037090452840776718
115+ },
116+ {
117+ "lambda": 0.3,
118+ "kappa": 0.0,
119+ "loss": 0.4298561159543437,
120+ "edits_per_token": 0.3309608540925267,
121+ "extra_ops_ratio": 0.09634915280596641,
122+ "loss_reduction": 0.0012070598127154608,
123+ "reduction_per_ops": 0.012527975364208016
124+ },
125+ {
126+ "lambda": 0.3,
127+ "kappa": 0.5,
128+ "loss": 0.4300144485165923,
129+ "edits_per_token": 0.11684460260972716,
130+ "extra_ops_ratio": 0.034015740434752945,
131+ "loss_reduction": 0.001048727250466841,
132+ "reduction_per_ops": 0.03083064596163796
133+ }
134+ ],
135+ "baseline_budget0": {
136+ "loss": 0.43106317576705916,
137+ "edits_per_token": 0.0,
138+ "extra_ops_ratio": 0.0
139+ },
140+ "selected": {
141+ "lambda": 0.3,
142+ "kappa": 0.5,
143+ "loss": 0.4300144485165923,
144+ "edits_per_token": 0.11684460260972716,
145+ "extra_ops_ratio": 0.034015740434752945,
146+ "loss_reduction": 0.001048727250466841,
147+ "reduction_per_ops": 0.03083064596163796
148+ },
149+ "checkpoint_sha256": "64570e628212ba370db86265ffaa23f655bfee6e58120829c5bbe5881c7aadd9",
150+ "selection_rule": "max_loss_reduction_per_ops"
18 } 151 }
\ No newline at end of file
added experiments/runs/controller_v2s1_seed01_20261030005758/checkpoint/checkpoint.json +49 −0
@@ -0,0 +1,49 @@
1+{
2+ "config": {
3+ "name": "tiny_p4v2",
4+ "vocab_size": 128,
5+ "d_model": 256,
6+ "n_layers": 6,
7+ "n_heads": 8,
8+ "n_kv_heads": 4,
9+ "ffn_width": 768,
10+ "max_seq_len": 320,
11+ "rope_theta": 10000.0,
12+ "norm_eps": 1e-05,
13+ "tie_embeddings": true,
14+ "init_std": 0.02,
15+ "use_workspace": true,
16+ "w": 64,
17+ "ffn_w": 192,
18+ "n_slots": 16,
19+ "n_editor_blocks": 2,
20+ "n_editor_heads": 2,
21+ "r_max": 8,
22+ "eta_max": 0.25,
23+ "clip_c": 1.0,
24+ "clip_eps": 1e-06,
25+ "readout_zero_init": false,
26+ "readout_init_std": 0.2,
27+ "readout_scale": 38.47460174560547,
28+ "controller_hidden": 0,
29+ "ctrl_input": "h",
30+ "ctrl_use_entropy": true,
31+ "lambda_cost": 0.3,
32+ "kappa_risk": 0.5,
33+ "dense_writes": false
34+ },
35+ "saved_at": "2026-10-30T01:00:19-0400",
36+ "run_id": "controller_v2s1_seed01_20261030005758",
37+ "parent": "experiments/runs/p4v2_prisme_seed01_20261030001749/checkpoint",
38+ "selected": {
39+ "lambda": 0.3,
40+ "kappa": 0.5,
41+ "loss": 0.48617894680615986,
42+ "edits_per_token": 0.15361803084223014,
43+ "extra_ops_ratio": 0.016404977543356864,
44+ "loss_reduction": 0.001227108467798943,
45+ "reduction_per_ops": 0.07480098430831782
46+ },
47+ "locked": true,
48+ "weights_sha256": "61a4dda3056e6047dd0671e53f2bb195aaa6d90f3fc112cc93fcf80d84ba4198"
49+}
\ No newline at end of file
added experiments/runs/controller_v2s1_seed01_20261030005758/config.yaml +67 −0
@@ -0,0 +1,67 @@
1+experiment:
2+ name: controller_v2s1
3+ seed: 1
4+ mode: prisme
5+ model: from_checkpoint
6+ checkpoint: experiments/runs/p4v2_prisme_seed01_20261030001749/checkpoint
7+ labels:
8+ train: experiments/runs/labels_v2s1_seed01_20261030005712/labels_train.pt
9+ calib: experiments/runs/labels_v2s1_seed01_20261030005712/labels_calib.pt
10+ train:
11+ steps: 2000
12+ lr: 0.001
13+ weight_decay: 0.01
14+ alpha: 1.0
15+ beta: 0.5
16+ gamma: 0.1
17+ huber_delta: 0.1
18+ log_every: 100
19+ eval_every: 250
20+ seq_len: 320
21+ lambda_grid:
22+ - 0.0
23+ - 0.01
24+ - 0.03
25+ - 0.1
26+ - 0.3
27+ kappa_grid:
28+ - 0.0
29+ - 0.5
30+ sweep_batches: 10
31+ max_budget: 8
32+ selection_rule: max_loss_reduction_per_ops
33+ model_overrides:
34+ ctrl_input: h
35+ ctrl_use_entropy: true
36+model:
37+ name: tiny_p4v2
38+ vocab_size: 128
39+ d_model: 256
40+ n_layers: 6
41+ n_heads: 8
42+ n_kv_heads: 4
43+ ffn_width: 768
44+ max_seq_len: 320
45+ rope_theta: 10000.0
46+ norm_eps: 1.0e-05
47+ tie_embeddings: true
48+ init_std: 0.02
49+ use_workspace: true
50+ w: 64
51+ ffn_w: 192
52+ n_slots: 16
53+ n_editor_blocks: 2
54+ n_editor_heads: 2
55+ r_max: 8
56+ eta_max: 0.25
57+ clip_c: 1.0
58+ clip_eps: 1.0e-06
59+ readout_zero_init: false
60+ readout_init_std: 0.2
61+ readout_scale: 38.47460174560547
62+ controller_hidden: 0
63+ ctrl_input: h
64+ ctrl_use_entropy: true
65+ lambda_cost: 0.0
66+ kappa_risk: 0.0
67+ dense_writes: false
added experiments/runs/controller_v2s1_seed01_20261030005758/manifest.json +151 −0
@@ -0,0 +1,151 @@
1+{
2+ "run_id": "controller_v2s1_seed01_20261030005758",
3+ "name": "controller_v2s1",
4+ "seed": 1,
5+ "code_commit": "unknown",
6+ "config_sha256": "1b270868a323af3e33c1bf1c047828f7743cb5844138936bbaee52634435e230",
7+ "started_at": "2026-10-30T00:57:58-0400",
8+ "host": "MacStudonpierre",
9+ "python": "3.12.14",
10+ "torch": "2.15.0",
11+ "command": "/Users/simon-pierreboucher/wp31/prisme-slm/training/train_controller.py --config configs/experiments/controller_v2s1.yaml --device mps",
12+ "state": "completed",
13+ "wall_seconds": 141.05,
14+ "device": "mps",
15+ "checkpoint_parent": "experiments/runs/p4v2_prisme_seed01_20261030001749/checkpoint",
16+ "labels_train": "experiments/runs/labels_v2s1_seed01_20261030005712/labels_train.pt",
17+ "labels_calib": "experiments/runs/labels_v2s1_seed01_20261030005712/labels_calib.pt",
18+ "finished_at": "2026-10-30T01:00:19-0400",
19+ "eval_train": {
20+ "huber": 0.0011783403564361462,
21+ "sign_acc": 0.6147150721618807,
22+ "harm_brier": 0.15530524830647194,
23+ "harm_ece": 0.031277772039175034,
24+ "n_labels": 26884,
25+ "oracle_opportunity_per_state": 0.01902814696367219,
26+ "regret_per_state": 0.01239542852347641,
27+ "captured_fraction": 0.3485740599365095,
28+ "top1_match": 0.06288532675709001,
29+ "n_audit_states": 1622
30+ },
31+ "eval_calib": {
32+ "huber": 0.0011220869855551932,
33+ "sign_acc": 0.6217491454896715,
34+ "harm_brier": 0.15860497322598402,
35+ "harm_ece": 0.02261553332209587,
36+ "n_labels": 26916,
37+ "oracle_opportunity_per_state": 0.01836320621989056,
38+ "regret_per_state": 0.010739420540064664,
39+ "captured_fraction": 0.41516637065089457,
40+ "top1_match": 0.06504559270516717,
41+ "n_audit_states": 1645
42+ },
43+ "sweep": [
44+ {
45+ "lambda": 0.0,
46+ "kappa": 0.0,
47+ "loss": 0.4843786382166092,
48+ "edits_per_token": 3.912811387900356,
49+ "extra_ops_ratio": 0.49989572029521034,
50+ "loss_reduction": 0.0030274170573496018,
51+ "reduction_per_ops": 0.00605609717074949
52+ },
53+ {
54+ "lambda": 0.0,
55+ "kappa": 0.5,
56+ "loss": 0.478396611422972,
57+ "edits_per_token": 2.082443653618031,
58+ "extra_ops_ratio": 0.24916935193298267,
59+ "loss_reduction": 0.009009443850986787,
60+ "reduction_per_ops": 0.036157913407464305
61+ },
62+ {
63+ "lambda": 0.01,
64+ "kappa": 0.0,
65+ "loss": 0.48403411629926835,
66+ "edits_per_token": 3.8208778173190985,
67+ "extra_ops_ratio": 0.4855178864683828,
68+ "loss_reduction": 0.0033719389746904582,
69+ "reduction_per_ops": 0.0069450355355961555
70+ },
71+ {
72+ "lambda": 0.01,
73+ "kappa": 0.5,
74+ "loss": 0.47863737855234745,
75+ "edits_per_token": 1.9727164887307236,
76+ "extra_ops_ratio": 0.23549998838847783,
77+ "loss_reduction": 0.008768676721611357,
78+ "reduction_per_ops": 0.03723429789366553
79+ },
80+ {
81+ "lambda": 0.03,
82+ "kappa": 0.0,
83+ "loss": 0.4810810790508809,
84+ "edits_per_token": 3.6109134045077105,
85+ "extra_ops_ratio": 0.4554090081302014,
86+ "loss_reduction": 0.006324976223077927,
87+ "reduction_per_ops": 0.013888561952357378
88+ },
89+ {
90+ "lambda": 0.03,
91+ "kappa": 0.5,
92+ "loss": 0.47920335201747616,
93+ "edits_per_token": 1.7520759193357058,
94+ "extra_ops_ratio": 0.20777568952893022,
95+ "loss_reduction": 0.008202703256482646,
96+ "reduction_per_ops": 0.03947864774305331
97+ },
98+ {
99+ "lambda": 0.1,
100+ "kappa": 0.0,
101+ "loss": 0.47742267803223815,
102+ "edits_per_token": 2.8718861209964412,
103+ "extra_ops_ratio": 0.35113779930033295,
104+ "loss_reduction": 0.009983377241720659,
105+ "reduction_per_ops": 0.028431508261466718
106+ },
107+ {
108+ "lambda": 0.1,
109+ "kappa": 0.5,
110+ "loss": 0.48005905920641845,
111+ "edits_per_token": 1.0676156583629892,
112+ "extra_ops_ratio": 0.1214192702676902,
113+ "loss_reduction": 0.007346996067540357,
114+ "reduction_per_ops": 0.06050930837701963
115+ },
116+ {
117+ "lambda": 0.3,
118+ "kappa": 0.0,
119+ "loss": 0.47899278570607323,
120+ "edits_per_token": 1.0723606168446027,
121+ "extra_ops_ratio": 0.11869999069901689,
122+ "loss_reduction": 0.008413269567885573,
123+ "reduction_per_ops": 0.07087843493786604
124+ },
125+ {
126+ "lambda": 0.3,
127+ "kappa": 0.5,
128+ "loss": 0.48617894680615986,
129+ "edits_per_token": 0.15361803084223014,
130+ "extra_ops_ratio": 0.016404977543356864,
131+ "loss_reduction": 0.001227108467798943,
132+ "reduction_per_ops": 0.07480098430831782
133+ }
134+ ],
135+ "baseline_budget0": {
136+ "loss": 0.4874060552739588,
137+ "edits_per_token": 0.0,
138+ "extra_ops_ratio": 0.0
139+ },
140+ "selected": {
141+ "lambda": 0.3,
142+ "kappa": 0.5,
143+ "loss": 0.48617894680615986,
144+ "edits_per_token": 0.15361803084223014,
145+ "extra_ops_ratio": 0.016404977543356864,
146+ "loss_reduction": 0.001227108467798943,
147+ "reduction_per_ops": 0.07480098430831782
148+ },
149+ "checkpoint_sha256": "61a4dda3056e6047dd0671e53f2bb195aaa6d90f3fc112cc93fcf80d84ba4198",
150+ "selection_rule": "max_loss_reduction_per_ops"
151+}
\ No newline at end of file
added experiments/runs/controller_v2s1_seed01_20261030005758/metrics.jsonl +30 −0
@@ -0,0 +1,30 @@
1+{"step": 0, "loss": 0.6217543482780457, "gain": 0.11268353462219238, "harm": 0.8242179155349731, "nll": 0.969618558883667, "t": 1.51}
2+{"step": 100, "loss": 0.19241824746131897, "gain": 0.003979177679866552, "harm": 0.5136688351631165, "nll": -0.6839534044265747, "t": 8.68}
3+{"step": 200, "loss": 0.05916588008403778, "gain": 0.0034271397162228823, "harm": 0.5102487206459045, "nll": -1.9938561916351318, "t": 14.33}
4+{"step": 249, "calib_huber": 0.010517006714784439, "calib_sign_acc": 0.6941967602912765, "calib_harm_brier": 0.1939330946408829, "calib_harm_ece": 0.07769986987113953, "calib_n_labels": 26916, "calib_oracle_opportunity_per_state": 0.01836320621989056, "calib_regret_per_state": 0.015281991320902816, "calib_captured_fraction": 0.16779286046737585, "calib_top1_match": 0.05410334346504559, "calib_n_audit_states": 1645, "t": 18.79}
5+{"step": 300, "loss": 0.14486049115657806, "gain": 0.003396645188331604, "harm": 0.6058646440505981, "nll": -1.6146845817565918, "t": 21.57}
6+{"step": 400, "loss": 0.07704775035381317, "gain": 0.0020318725146353245, "harm": 0.5852528810501099, "nll": -2.176105499267578, "t": 26.95}
7+{"step": 499, "calib_huber": 0.0015250327621141929, "calib_sign_acc": 0.5629365433199583, "calib_harm_brier": 0.17094168093994974, "calib_harm_ece": 0.022884827107191086, "calib_n_labels": 26916, "calib_oracle_opportunity_per_state": 0.01836320621989056, "calib_regret_per_state": 0.01282272375101014, "calib_captured_fraction": 0.30171650868240585, "calib_top1_match": 0.0547112462006079, "calib_n_audit_states": 1645, "t": 33.86}
8+{"step": 500, "loss": 0.001370161771774292, "gain": 0.001527803367935121, "harm": 0.5307777523994446, "nll": -2.6554653644561768, "t": 33.92}
9+{"step": 600, "loss": 0.10702954232692719, "gain": 0.006142090540379286, "harm": 0.4747239649295807, "nll": -1.3647453784942627, "t": 39.36}
10+{"step": 700, "loss": -0.010433346033096313, "gain": 0.0026992247439920902, "harm": 0.42587319016456604, "nll": -2.2606916427612305, "t": 44.74}
11+{"step": 749, "calib_huber": 0.0016405455295319259, "calib_sign_acc": 0.4800861940853024, "calib_harm_brier": 0.16535962802514506, "calib_harm_ece": 0.01039370521903038, "calib_n_labels": 26916, "calib_oracle_opportunity_per_state": 0.01836320621989056, "calib_regret_per_state": 0.012461327057116662, "calib_captured_fraction": 0.3213969876557359, "calib_top1_match": 0.0547112462006079, "calib_n_audit_states": 1645, "t": 49.04}
12+{"step": 800, "loss": -0.019962161779403687, "gain": 0.0013843975029885769, "harm": 0.48733922839164734, "nll": -2.6501615047454834, "t": 51.84}
13+{"step": 900, "loss": -0.058852121233940125, "gain": 0.0009405212476849556, "harm": 0.4586194157600403, "nll": -2.8910233974456787, "t": 57.39}
14+{"step": 999, "calib_huber": 0.0012022649852991068, "calib_sign_acc": 0.6564125427255164, "calib_harm_brier": 0.1600398019582879, "calib_harm_ece": 0.01565980352461338, "calib_n_labels": 26916, "calib_oracle_opportunity_per_state": 0.01836320621989056, "calib_regret_per_state": 0.011427309013065233, "calib_captured_fraction": 0.3777062199145015, "calib_top1_match": 0.06322188449848025, "calib_n_audit_states": 1645, "t": 64.34}
15+{"step": 1000, "loss": -0.04631441831588745, "gain": 0.0013401865726336837, "harm": 0.4754399061203003, "nll": -2.853745460510254, "t": 64.4}
16+{"step": 1100, "loss": -0.07637041807174683, "gain": 0.0010725471656769514, "harm": 0.4387507736682892, "nll": -2.9681835174560547, "t": 69.79}
17+{"step": 1200, "loss": -0.06496578454971313, "gain": 0.0010426120134070516, "harm": 0.4698534905910492, "nll": -3.0093514919281006, "t": 75.21}
18+{"step": 1249, "calib_huber": 0.0011220869855551932, "calib_sign_acc": 0.6217491454896715, "calib_harm_brier": 0.15860497322598402, "calib_harm_ece": 0.02261553332209587, "calib_n_labels": 26916, "calib_oracle_opportunity_per_state": 0.01836320621989056, "calib_regret_per_state": 0.010739420540064664, "calib_captured_fraction": 0.41516637065089457, "calib_top1_match": 0.06504559270516717, "calib_n_audit_states": 1645, "t": 79.37}
19+{"step": 1300, "loss": -0.1207217276096344, "gain": 0.0005600811564363539, "harm": 0.4300370216369629, "nll": -3.3630030155181885, "t": 82.05}
20+{"step": 1400, "loss": -0.04722009599208832, "gain": 0.0011644494952633977, "harm": 0.45957231521606445, "nll": -2.7817070484161377, "t": 87.3}
21+{"step": 1499, "calib_huber": 0.0009785579579212032, "calib_sign_acc": 0.42041908158716007, "calib_harm_brier": 0.16143824849020302, "calib_harm_ece": 0.013009194284677505, "calib_n_labels": 26916, "calib_oracle_opportunity_per_state": 0.01836320621989056, "calib_regret_per_state": 0.01221910088257949, "calib_captured_fraction": 0.3345878308906606, "calib_top1_match": 0.06869300911854104, "calib_n_audit_states": 1645, "t": 94.15}
22+{"step": 1500, "loss": -0.06804871559143066, "gain": 0.000664825551211834, "harm": 0.5305606126785278, "nll": -3.3399384021759033, "t": 94.21}
23+{"step": 1600, "loss": -0.07556450366973877, "gain": 0.001154533470980823, "harm": 0.5068498849868774, "nll": -3.3014397621154785, "t": 99.54}
24+{"step": 1700, "loss": -0.10204572975635529, "gain": 0.001108089811168611, "harm": 0.4906119704246521, "nll": -3.48459792137146, "t": 104.94}
25+{"step": 1749, "calib_huber": 0.0009797979088112332, "calib_sign_acc": 0.6669638876504681, "calib_harm_brier": 0.16354970610288794, "calib_harm_ece": 0.049120619893074036, "calib_n_labels": 26916, "calib_oracle_opportunity_per_state": 0.01836320621989056, "calib_regret_per_state": 0.012027452854399986, "calib_captured_fraction": 0.34502435411457977, "calib_top1_match": 0.08875379939209727, "calib_n_audit_states": 1645, "t": 109.14}
26+{"step": 1800, "loss": -0.11715850234031677, "gain": 0.0008175690891221166, "harm": 0.4343639016151428, "nll": -3.3515801429748535, "t": 111.91}
27+{"step": 1900, "loss": -0.1207641214132309, "gain": 0.0008810321451164782, "harm": 0.4911242127418518, "nll": -3.672072649002075, "t": 117.51}
28+{"step": 1999, "loss": -0.11400634050369263, "gain": 0.0009010079083964229, "harm": 0.47006553411483765, "nll": -3.499401092529297, "t": 122.79}
29+{"step": 1999, "calib_huber": 0.0009146865885242485, "calib_sign_acc": 0.4517387427552385, "calib_harm_brier": 0.15814475145820417, "calib_harm_ece": 0.018627993762493134, "calib_n_labels": 26916, "calib_oracle_opportunity_per_state": 0.01836320621989056, "calib_regret_per_state": 0.01151024432892495, "calib_captured_fraction": 0.37318983454766497, "calib_top1_match": 0.07963525835866261, "calib_n_audit_states": 1645, "t": 124.31}
30+{"event": "restored_best_calibration_checkpoint", "step": 1250, "score": -0.010739420540064664, "t": 124.31}
added experiments/runs/labels_v2s1_seed01_20261030005712/config.yaml +58 −0
@@ -0,0 +1,58 @@
1+experiment:
2+ name: labels_v2s1
3+ seed: 1
4+ mode: prisme
5+ model: from_checkpoint
6+ checkpoint: experiments/runs/p4v2_prisme_seed01_20261030001749/checkpoint
7+ use_policy: false
8+ data:
9+ kind: synthetic
10+ tokenizer: char
11+ splits_dir: data/splits/synthetic_v1
12+ loss_span: answer
13+ collect:
14+ batch_size: 32
15+ seq_len: 320
16+ max_prefix: 3
17+ audit_every: 4
18+ partitions:
19+ train:
20+ split: ctrl_train
21+ n_batches: 300
22+ stream_seed: 11
23+ calib:
24+ split: calib
25+ n_batches: 40
26+ stream_seed: 12
27+model:
28+ name: tiny_p4v2
29+ vocab_size: 128
30+ d_model: 256
31+ n_layers: 6
32+ n_heads: 8
33+ n_kv_heads: 4
34+ ffn_width: 768
35+ max_seq_len: 320
36+ rope_theta: 10000.0
37+ norm_eps: 1.0e-05
38+ tie_embeddings: true
39+ init_std: 0.02
40+ use_workspace: true
41+ w: 64
42+ ffn_w: 192
43+ n_slots: 16
44+ n_editor_blocks: 2
45+ n_editor_heads: 2
46+ r_max: 8
47+ eta_max: 0.25
48+ clip_c: 1.0
49+ clip_eps: 1.0e-06
50+ readout_zero_init: false
51+ readout_init_std: 0.2
52+ readout_scale: 38.47460174560547
53+ controller_hidden: 0
54+ ctrl_input: z
55+ ctrl_use_entropy: false
56+ lambda_cost: 0.0
57+ kappa_risk: 0.0
58+ dense_writes: false
added experiments/runs/labels_v2s1_seed01_20261030005712/manifest.json +40 −0
@@ -0,0 +1,40 @@
1+{
2+ "run_id": "labels_v2s1_seed01_20261030005712",
3+ "name": "labels_v2s1",
4+ "seed": 1,
5+ "code_commit": "unknown",
6+ "config_sha256": "d8a7c72237e220d4d2c97a848283a4500bd5af90fdd38e416f3c63cbf99d54af",
7+ "started_at": "2026-10-30T00:57:12-0400",
8+ "host": "MacStudonpierre",
9+ "python": "3.12.14",
10+ "torch": "2.15.0",
11+ "command": "/Users/simon-pierreboucher/wp31/prisme-slm/training/collect_interventions.py --config configs/experiments/labels_v2s1.yaml --device mps",
12+ "state": "completed",
13+ "wall_seconds": 44.65,
14+ "device": "mps",
15+ "checkpoint_parent": "experiments/runs/p4v2_prisme_seed01_20261030001749/checkpoint",
16+ "checkpoint_sha256": "63501504cc2f44e41c51c3c2ec941160a997e2e5035167f0cf79e8de7dc00d37",
17+ "finished_at": "2026-10-30T00:57:57-0400",
18+ "partitions": {
19+ "train": {
20+ "file": "experiments/runs/labels_v2s1_seed01_20261030005712/labels_train.pt",
21+ "n_states": 49965,
22+ "n_labels": 199860,
23+ "mean_gain": 0.004881912376731634,
24+ "frac_positive": 0.3325177729129791,
25+ "frac_harm": 0.5166516304016113,
26+ "seconds": 39.4,
27+ "n_audit_batches": 75
28+ },
29+ "calib": {
30+ "file": "experiments/runs/labels_v2s1_seed01_20261030005712/labels_calib.pt",
31+ "n_states": 6729,
32+ "n_labels": 26916,
33+ "mean_gain": 0.005064019002020359,
34+ "frac_positive": 0.33912914991378784,
35+ "frac_harm": 0.5117030739784241,
36+ "seconds": 5.2,
37+ "n_audit_batches": 10
38+ }
39+ }
40+}
\ No newline at end of file
added experiments/runs/labels_v2s1_seed01_20261030005712/metrics.jsonl +0 −0
modified paper/appendix/failures.tex +2 −0
@@ -18,4 +18,6 @@
18 \textbf{date}: 2026-10-30; \textbf{what failed}: v2 tiny backbone (5.6M, 15k planned steps) overfit the 23k-item train split: validation loss (all tokens) 0.246 at step 6k $\rightarrow$ 0.282 at step 9k. Run stopped at step 9,300; the step-5,000 checkpoint (val 0.249) is used as the frozen v2 backbone.; \textbf{evidence}: \texttt{experiments/logs/p4\_tiny\_backbone\_v2.log}, run \texttt{p4\_tiny\_backbone\_v2\_seed01\_*} (state: stopped); \textbf{resolution}: 42 epochs of a small synthetic set is too many; schedule not re-run (the checkpoint is adequate as a frozen comparator). 18 \textbf{date}: 2026-10-30; \textbf{what failed}: v2 tiny backbone (5.6M, 15k planned steps) overfit the 23k-item train split: validation loss (all tokens) 0.246 at step 6k $\rightarrow$ 0.282 at step 9k. Run stopped at step 9,300; the step-5,000 checkpoint (val 0.249) is used as the frozen v2 backbone.; \textbf{evidence}: \texttt{experiments/logs/p4\_tiny\_backbone\_v2.log}, run \texttt{p4\_tiny\_backbone\_v2\_seed01\_*} (state: stopped); \textbf{resolution}: 42 epochs of a small synthetic set is too many; schedule not re-run (the checkpoint is adequate as a frozen comparator).
19 \item 19 \item
20 \textbf{date}: 2026-10-30; \textbf{what failed}: Killed to free nodes for the 5-seed matrix: the degraded joint-finetune reference (M4M64a) and the v1 dense-writes run on the slow m4mn node (superseded by the v2 matrix \texttt{p4v2\_dense}).; \textbf{evidence}: manifests remain in \texttt{experiments/runs/} on the nodes (state: running $\rightarrow$ killed) 20 \textbf{date}: 2026-10-30; \textbf{what failed}: Killed to free nodes for the 5-seed matrix: the degraded joint-finetune reference (M4M64a) and the v1 dense-writes run on the slow m4mn node (superseded by the v2 matrix \texttt{p4v2\_dense}).; \textbf{evidence}: manifests remain in \texttt{experiments/runs/} on the nodes (state: running $\rightarrow$ killed)
21+\item
22+ \textbf{date}: 2026-10-30; \textbf{what failed}: Ranking-loss controller (D12: pairwise + listwise on top of Huber/BCE/NLL, editor-unseen labels): calibration sign accuracy fell to 45 \%, captured fraction --8 \%; on dev-iid the locked policy ($\lambda$=0.3, $\kappa$=0.5) edits 0.07 tokens and matches budget 0 (0.4333 vs 0.4334); $\lambda$=0 is worse (+1.3 \%).; \textbf{evidence}: \texttt{experiments/runs/controller\_v1rank\_seed01\_*}, \texttt{results/raw/routing\_v1rank.json}; \textbf{resolution}: Within-state ranking is not learnable from these labels at this scale; H3 null stands.
21 \end{itemize} 23 \end{itemize}
modified paper/main.log +19 −19
@@ -1,4 +1,4 @@
1−This is pdfTeX, Version 3.141592653-2.6-1.40.29 (TeX Live 2026) (preloaded format=pdflatex 2026.3.22) 30 OCT 2026 00:55 1+This is pdfTeX, Version 3.141592653-2.6-1.40.29 (TeX Live 2026) (preloaded format=pdflatex 2026.3.22) 30 OCT 2026 00:58
2 entering extended mode 2 entering extended mode
3 restricted \write18 enabled. 3 restricted \write18 enabled.
4 %&-line parsing enabled. 4 %&-line parsing enabled.
@@ -1319,22 +1319,22 @@ Overfull \hbox (133.05286pt too wide) in paragraph at lines 7--18
1319 [] 1319 []
1320 1320
1321 ) [16] [17 <../results/figures/learnability_bootstrap.pdf>] 1321 ) [16] [17 <../results/figures/learnability_bootstrap.pdf>]
1322−<../results/figures/labels_analysis.pdf, id=518, 478.87544pt x 181.0464pt> 1322+<../results/figures/labels_analysis.pdf, id=518, 548.77519pt x 192.17598pt>
1323 File: ../results/figures/labels_analysis.pdf Graphic file (type pdf) 1323 File: ../results/figures/labels_analysis.pdf Graphic file (type pdf)
1324 <use ../results/figures/labels_analysis.pdf> 1324 <use ../results/figures/labels_analysis.pdf>
1325 Package pdftex.def Info: ../results/figures/labels_analysis.pdf used on input 1325 Package pdftex.def Info: ../results/figures/labels_analysis.pdf used on input
1326 line 72. 1326 line 72.
1327−(pdftex.def) Requested size: 469.75502pt x 177.60933pt. 1327+(pdftex.def) Requested size: 469.75502pt x 164.50862pt.
1328 1328
1329 (./tables/routing.tex) (./tables/controller.tex) (./tables/routing_seeds.tex 1329 (./tables/routing.tex) (./tables/controller.tex) (./tables/routing_seeds.tex
1330 Overfull \hbox (109.74734pt too wide) in paragraph at lines 8--14 1330 Overfull \hbox (109.74734pt too wide) in paragraph at lines 8--14
1331 [][] 1331 [][]
1332 [] 1332 []
1333 1333
1334−) [18] [19 <../results/figures/labels_analysis.pdf>] (./tables/latency.tex) 1334+) [18] (./tables/stats.tex) [19 <../results/figures/labels_analysis.pdf>]
1335−(./sections/results_pilot.tex)) (./sections/mechanistic.tex [20]) 1335+(./tables/latency.tex) (./sections/results_pilot.tex))
1336−(./sections/limitations.tex [21]) (./sections/conclusion.tex) 1336+(./sections/mechanistic.tex [20]) (./sections/limitations.tex [21])
1337−(./sections/reproducibility.tex) 1337+(./sections/conclusion.tex) (./sections/reproducibility.tex [22])
1338 Overfull \hbox (51.18217pt too wide) in paragraph at lines 2--101 1338 Overfull \hbox (51.18217pt too wide) in paragraph at lines 2--101
1339 \T1/ptm/m/n/12 (-20) num-ber in the manuscript is pro-duced by \T1/cmtt/m/n/12 1339 \T1/ptm/m/n/12 (-20) num-ber in the manuscript is pro-duced by \T1/cmtt/m/n/12
1340 scripts/make_tables.py \T1/ptm/m/n/12 (-20) and \T1/cmtt/m/n/12 scripts/reprodu 1340 scripts/make_tables.py \T1/ptm/m/n/12 (-20) and \T1/cmtt/m/n/12 scripts/reprodu
@@ -1348,7 +1348,7 @@ cript_claims.py \T1/ptm/m/n/12 (-20) checks the chain. \T1/cmtt/m/n/12 REPRODUC
1348 E.md 1348 E.md
1349 [] 1349 []
1350 1350
1351−[22] [23] (./main.bbl [24] [25] 1351+[23] (./main.bbl [24] [25]
1352 Underfull \hbox (badness 1297) in paragraph at lines 208--211 1352 Underfull \hbox (badness 1297) in paragraph at lines 208--211
1353 []\T1/ptm/b/n/12 (+20) You, Yichen, Tianyu Fu, Aosong Feng, Xing-tai Lv, Xue-fe 1353 []\T1/ptm/b/n/12 (+20) You, Yichen, Tianyu Fu, Aosong Feng, Xing-tai Lv, Xue-fe
1354 i Ning, Ning Ding, and 1354 i Ning, Ning Ding, and
@@ -1561,7 +1561,7 @@ run \T1/cmtt/m/n/12 p4_tiny_backbone_v2_seed01_* \T1/ptm/m/n/12 (-20) (state:
1561 [] 1561 []
1562 1562
1563 )) [35{/usr/local/texlive/2026/texmf-dist/fonts/enc/dvips/cm-super/cm-super-ts1 1563 )) [35{/usr/local/texlive/2026/texmf-dist/fonts/enc/dvips/cm-super/cm-super-ts1
1564−.enc}] [36] [37] [38] (./main.aux) 1564+.enc}] [36] [37] [38] [39] (./main.aux)
1565 *********** 1565 ***********
1566 LaTeX2e <2025-11-01> 1566 LaTeX2e <2025-11-01>
1567 L3 programming layer <2026-01-19> 1567 L3 programming layer <2026-01-19>
@@ -1571,14 +1571,14 @@ L3 programming layer <2026-01-19>
1571 LaTeX Warning: There were undefined references. 1571 LaTeX Warning: There were undefined references.
1572 1572
1573 Package rerunfilecheck Info: File `main.out' has not changed. 1573 Package rerunfilecheck Info: File `main.out' has not changed.
1574−(rerunfilecheck) Checksum: 09A4B2F3436223C5D37E56F8E45DB6B6;9020. 1574+(rerunfilecheck) Checksum: A0F1ADB953B7C9C76AC4BF5EDF00287D;9020.
1575 ) 1575 )
1576 Here is how much of TeX's memory you used: 1576 Here is how much of TeX's memory you used:
1577− 36051 strings out of 467525 1577+ 36063 strings out of 467525
1578− 701004 string characters out of 5418982 1578+ 701177 string characters out of 5418982
1579− 1125706 words of memory out of 5000000 1579+ 1129055 words of memory out of 5000000
1580− 63382 multiletter control sequences out of 15000+600000 1580+ 63386 multiletter control sequences out of 15000+600000
1581− 793071 words of font info for 661 fonts, out of 8000000 for 9000 1581+ 793198 words of font info for 663 fonts, out of 8000000 for 9000
1582 1141 hyphenation exceptions out of 8191 1582 1141 hyphenation exceptions out of 8191
1583 84i,13n,104p,1001b,1501s stack positions out of 10000i,1000n,20000p,200000b,200000s 1583 84i,13n,104p,1001b,1501s stack positions out of 10000i,1000n,20000p,200000b,200000s
1584 </usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmex10.pfb 1584 </usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmex10.pfb
@@ -1596,10 +1596,10 @@ exlive/2026/texmf-dist/fonts/type1/urw/symbol/usyr.pfb></usr/local/texlive/2026
1596 e1/urw/times/utmbi8a.pfb></usr/local/texlive/2026/texmf-dist/fonts/type1/urw/ti 1596 e1/urw/times/utmbi8a.pfb></usr/local/texlive/2026/texmf-dist/fonts/type1/urw/ti
1597 mes/utmr8a.pfb></usr/local/texlive/2026/texmf-dist/fonts/type1/urw/times/utmri8 1597 mes/utmr8a.pfb></usr/local/texlive/2026/texmf-dist/fonts/type1/urw/times/utmri8
1598 a.pfb> 1598 a.pfb>
1599−Output written on main.pdf (38 pages, 415081 bytes). 1599+Output written on main.pdf (39 pages, 416998 bytes).
1600 PDF statistics: 1600 PDF statistics:
1601− 759 PDF objects out of 1000 (max. 8388607) 1601+ 764 PDF objects out of 1000 (max. 8388607)
1602− 659 compressed objects within 7 object streams 1602+ 663 compressed objects within 7 object streams
1603− 153 named destinations out of 1000 (max. 500000) 1603+ 155 named destinations out of 1000 (max. 500000)
1604 143772 words of extra memory for PDF output out of 154059 (max. 10000000) 1604 143772 words of extra memory for PDF output out of 154059 (max. 10000000)
1605 1605
modified paper/main.pdf +0 −0

Binary file not shown.

modified paper/sections/results.tex +13 −4
@@ -103,15 +103,24 @@ over \vtwoctrlseeds{} seeds the locked controller changes the dev-iid loss by \v
103 \vtwoctrlrelsd{}) while editing a small fraction of tokens, the best fixed schedule lowers it by 103 \vtwoctrlrelsd{}) while editing a small fraction of tokens, the best fixed schedule lowers it by
104 \vtwobestfixedrel{}\,\% and the greedy oracle by \vtwooraclerel{}\,\%; the controller beats the frozen backbone 104 \vtwobestfixedrel{}\,\% and the greedy oracle by \vtwooraclerel{}\,\%; the controller beats the frozen backbone
105 on \vtwoctrlbetter{} of \vtwoctrlseeds{} seeds and never approaches the fixed schedules at any cost. 105 on \vtwoctrlbetter{} of \vtwoctrlseeds{} seeds and never approaches the fixed schedules at any cost.
106+(v) Adding within-state ranking losses (pairwise on the logged candidates, listwise on the exhaustive audits;
107+Appendix~\ref{app:failures}) did not change the picture: the locked policy edits 0.07 tokens and matches
108+budget 0 (0.4333 vs 0.4334), and every smaller penalty is worse.
106 \input{tables/routing.tex} 109 \input{tables/routing.tex}
107 \input{tables/controller.tex} 110 \input{tables/controller.tex}
108 \input{tables/routing_seeds.tex} 111 \input{tables/routing_seeds.tex}
109 112
110 \subsection{Token loss versus task success}\label{sec:results-reasoning} 113 \subsection{Token loss versus task success}\label{sec:results-reasoning}
111−Exact match of free greedy generation moves with the loss on dev-iid (Table~\ref{tab:seedmatrix}: $+1.8$ points 114+Exact match of free greedy generation moves with the loss on dev-iid but by less than the pre-registered
112−for the sparse editor at budget 2, $+2.3$ for global recurrence at budget 8, seed SD $\le 0.5$) but not on the 115+minimum effect of three points (Table~\ref{tab:stats}): the sparse editor gains $+1.9$ points at budget 2
113−held-out regimes; per-family transitions are reported in Appendix~\ref{app:results}. No adapter turns the 116+(95\,\% $t$-interval over five seeds $[+1.6,+2.2]$; every per-seed paired item bootstrap excludes zero), global
114−held-out compose family from near zero (2--3\,\%) into anything else. 117+recurrence gains $+1.9$ at budget 2 and $+2.3$ at budget 8, and the paired difference between the two systems
118+at budget 2 is $+0.1$ points $[-0.5,+0.7]$---a null---while at budget 1 global recurrence is ahead by $0.6$
119+points $[0.3,0.9]$. Transitions are two-sided: at budget 2 the sparse editor turns \vtwoprismetransplus{}
120+items per thousand from wrong to right and \vtwoprismetransminus{} from right to wrong
121+(Table~\ref{tab:emfamilies}); gains concentrate in \textsf{fsm} and \textsf{bind}, \textsf{excl} does not move.
122+No adapter changes the held-out regimes by more than a point, and the compose family stays at 2--3\,\%.
123+\input{tables/stats.tex}
115 124
116 \subsection{Latency and memory}\label{sec:results-latency} 125 \subsection{Latency and memory}\label{sec:results-latency}
117 Table~\ref{tab:latency} gives measured batch-1 and batch-8 decode latencies for the tiny model on the 126 Table~\ref{tab:latency} gives measured batch-1 and batch-8 decode latencies for the tiny model on the
modified paper/tables/numbers.tex +6 −6
@@ -57,12 +57,12 @@
57 \newcommand{\routinguniformbeight}{+0.6} 57 \newcommand{\routinguniformbeight}{+0.6}
58 \newcommand{\routinguniformbone}{-0.8} 58 \newcommand{\routinguniformbone}{-0.8}
59 \newcommand{\routinguniformbtwo}{-0.7} 59 \newcommand{\routinguniformbtwo}{-0.7}
60−\newcommand{\vtwobestfixedrel}{-4.0} 60+\newcommand{\vtwobestfixedrel}{-4.2}
61 \newcommand{\vtwobzeroloss}{0.4621} 61 \newcommand{\vtwobzeroloss}{0.4621}
62−\newcommand{\vtwoctrlbetter}{2} 62+\newcommand{\vtwoctrlbetter}{3}
63−\newcommand{\vtwoctrlrel}{+0.5} 63+\newcommand{\vtwoctrlrel}{+0.3}
64−\newcommand{\vtwoctrlrelsd}{1.1} 64+\newcommand{\vtwoctrlrelsd}{1.0}
65−\newcommand{\vtwoctrlseeds}{4} 65+\newcommand{\vtwoctrlseeds}{5}
66 \newcommand{\vtwodenseembeight}{n/a} 66 \newcommand{\vtwodenseembeight}{n/a}
67 \newcommand{\vtwodenseembtwo}{n/a} 67 \newcommand{\vtwodenseembtwo}{n/a}
68 \newcommand{\vtwodenseembzero}{n/a} 68 \newcommand{\vtwodenseembzero}{n/a}
@@ -81,7 +81,7 @@
81 \newcommand{\vtwogrseeds}{5} 81 \newcommand{\vtwogrseeds}{5}
82 \newcommand{\vtwogrtransminus}{39} 82 \newcommand{\vtwogrtransminus}{39}
83 \newcommand{\vtwogrtransplus}{57} 83 \newcommand{\vtwogrtransplus}{57}
84−\newcommand{\vtwooraclerel}{-15.0} 84+\newcommand{\vtwooraclerel}{-15.1}
85 \newcommand{\vtwoprismeembeight}{39.2} 85 \newcommand{\vtwoprismeembeight}{39.2}
86 \newcommand{\vtwoprismeembtwo}{39.6} 86 \newcommand{\vtwoprismeembtwo}{39.6}
87 \newcommand{\vtwoprismeembzero}{37.7} 87 \newcommand{\vtwoprismeembzero}{37.7}
modified paper/tables/routing_seeds.tex +1 −0
@@ -7,6 +7,7 @@ opportunity for each controller.}
7 \label{tab:routingseeds} 7 \label{tab:routingseeds}
8 \begin{tabular}{lrlrrrrrr}\toprule 8 \begin{tabular}{lrlrrrrrr}\toprule
9 seed & budget 0 & best fixed & entropy & norm & controller (edits) & oracle b=8 (edits) & sign acc. & captured\\\midrule 9 seed & budget 0 & best fixed & entropy & norm & controller (edits) & oracle b=8 (edits) & sign acc. & captured\\\midrule
10+1 & 0.4966 & 0.4738 (random forced b=4) & 0.4780 & 0.4774 & 0.4950 (0.18) & 0.4198 (2.16) & 0.622 & +0.415\\
10 2 & 0.4966 & 0.4754 (random forced b=4) & 0.4799 & 0.4794 & 0.4965 (0.05) & 0.4114 (1.90) & 0.413 & +0.333\\ 11 2 & 0.4966 & 0.4754 (random forced b=4) & 0.4799 & 0.4794 & 0.4965 (0.05) & 0.4114 (1.90) & 0.413 & +0.333\\
11 3 & 0.4966 & 0.4775 (uniform b=8) & 0.4825 & 0.4814 & 0.5074 (4.32) & 0.4242 (1.84) & 0.504 & +0.305\\ 12 3 & 0.4966 & 0.4775 (uniform b=8) & 0.4825 & 0.4814 & 0.5074 (4.32) & 0.4242 (1.84) & 0.504 & +0.305\\
12 4 & 0.4966 & 0.4762 (uniform b=8) & 0.4795 & 0.4790 & 0.4974 (0.17) & 0.4251 (2.19) & 0.633 & +0.521\\ 13 4 & 0.4966 & 0.4762 (uniform b=8) & 0.4795 & 0.4790 & 0.4974 (0.17) & 0.4251 (2.19) & 0.633 & +0.521\\
added paper/tables/stats.tex +17 −0
@@ -0,0 +1,17 @@
1+\begin{table}[t]\centering\small
2+\caption{Pre-registered uncertainty analysis of dev-iid exact match (percentage points). ``t-CI'' is the 95\,\%
3+$t$-interval over independent adapter seeds of the per-seed paired difference; ``item CI range'' spans the
4+smallest lower and largest upper bound of the per-seed paired item bootstraps (10,000 resamples, 1,000 items).}
5+\label{tab:stats}
6+\begin{tabular}{lrrll}\toprule
7+Comparison & seeds & mean & t-CI over seeds & item CI range\\\midrule
8+prisme b1 vs b0 & 5 & +1.58 & [+1.21, +1.95] & [+0.30, +3.30]\\
9+prisme b2 vs b0 & 5 & +1.92 & [+1.60, +2.24] & [+0.10, +3.90]\\
10+prisme b8 vs b0 & 5 & +1.50 & [+0.94, +2.06] & [-1.10, +4.50]\\
11+gr b1 vs b0 & 5 & +2.14 & [+1.76, +2.52] & [+0.20, +4.10]\\
12+gr b2 vs b0 & 5 & +1.86 & [+1.37, +2.35] & [-0.30, +4.50]\\
13+gr b8 vs b0 & 5 & +2.34 & [+0.97, +3.71] & [-1.40, +6.20]\\
14+prisme minus gr b1 & 5 & -0.56 & [-0.85, -0.27] & [-2.20, +0.90]\\
15+prisme minus gr b2 & 5 & +0.06 & [-0.53, +0.65] & [-2.20, +2.00]\\
16+prisme minus gr b8 & 5 & -0.84 & [-2.43, +0.75] & [-4.40, +2.50]\\
17+\bottomrule\end{tabular}\end{table}
modified results/analyzed/routing_seeds_v2.json +16 −0
@@ -1,4 +1,20 @@
1 [ 1 [
2+ {
3+ "seed": "1",
4+ "b0": 0.4965549189338985,
5+ "best_fixed": 0.4737940016007751,
6+ "best_fixed_name": "random forced b=4",
7+ "entropy": 0.47797408758425247,
8+ "norm": 0.4774494416747904,
9+ "ctrl": 0.4950156743532738,
10+ "ctrl_edits": 0.18027509511267192,
11+ "ctrl_lam": 0.3,
12+ "ctrl_kap": 0.5,
13+ "oracle8": 0.4197931946875673,
14+ "oracle8_edits": 2.1623277182235836,
15+ "sign_acc": 0.6217491454896715,
16+ "captured": 0.41516637065089457
17+ },
2 { 18 {
3 "seed": "2", 19 "seed": "2",
4 "b0": 0.49655490497905047, 20 "b0": 0.49655490497905047,
added results/analyzed/statistics_v2.json +506 −0
@@ -0,0 +1,506 @@
1+{
2+ "prisme_b1_vs_b0": {
3+ "per_seed": [
4+ {
5+ "diff": 0.021,
6+ "ci_low": 0.01,
7+ "ci_high": 0.033,
8+ "n": 1000,
9+ "p_two_sided": 0.0,
10+ "run_id": "p4v2_prisme_seed01_20261030001749"
11+ },
12+ {
13+ "diff": 0.014,
14+ "ci_low": 0.003,
15+ "ci_high": 0.025,
16+ "n": 1000,
17+ "p_two_sided": 0.0154,
18+ "run_id": "p4v2_prisme_seed02_20261030001749"
19+ },
20+ {
21+ "diff": 0.014,
22+ "ci_low": 0.003,
23+ "ci_high": 0.026,
24+ "n": 1000,
25+ "p_two_sided": 0.0182,
26+ "run_id": "p4v2_prisme_seed03_20261030001749"
27+ },
28+ {
29+ "diff": 0.015,
30+ "ci_low": 0.004,
31+ "ci_high": 0.026,
32+ "n": 1000,
33+ "p_two_sided": 0.0104,
34+ "run_id": "p4v2_prisme_seed04_20261030001749"
35+ },
36+ {
37+ "diff": 0.015,
38+ "ci_low": 0.004,
39+ "ci_high": 0.027,
40+ "n": 1000,
41+ "p_two_sided": 0.0098,
42+ "run_id": "p4v2_prisme_seed05_20261030001839"
43+ }
44+ ],
45+ "seeds": {
46+ "mean": 0.0158,
47+ "sd": 0.0029495762407505256,
48+ "ci_low": 0.012137617371902876,
49+ "ci_high": 0.019462382628097127,
50+ "n": 5
51+ }
52+ },
53+ "prisme_b2_vs_b0": {
54+ "per_seed": [
55+ {
56+ "diff": 0.023,
57+ "ci_low": 0.008,
58+ "ci_high": 0.039,
59+ "n": 1000,
60+ "p_two_sided": 0.0042,
61+ "run_id": "p4v2_prisme_seed01_20261030001749"
62+ },
63+ {
64+ "diff": 0.016,
65+ "ci_low": 0.001,
66+ "ci_high": 0.031,
67+ "n": 1000,
68+ "p_two_sided": 0.0416,
69+ "run_id": "p4v2_prisme_seed02_20261030001749"
70+ },
71+ {
72+ "diff": 0.02,
73+ "ci_low": 0.005,
74+ "ci_high": 0.035,
75+ "n": 1000,
76+ "p_two_sided": 0.0112,
77+ "run_id": "p4v2_prisme_seed03_20261030001749"
78+ },
79+ {
80+ "diff": 0.018,
81+ "ci_low": 0.004,
82+ "ci_high": 0.032,
83+ "n": 1000,
84+ "p_two_sided": 0.0124,
85+ "run_id": "p4v2_prisme_seed04_20261030001749"
86+ },
87+ {
88+ "diff": 0.019,
89+ "ci_low": 0.004,
90+ "ci_high": 0.035,
91+ "n": 1000,
92+ "p_two_sided": 0.0136,
93+ "run_id": "p4v2_prisme_seed05_20261030001839"
94+ }
95+ ],
96+ "seeds": {
97+ "mean": 0.019200000000000002,
98+ "sd": 0.002588435821108957,
99+ "ci_low": 0.01598603242926801,
100+ "ci_high": 0.022413967570731993,
101+ "n": 5
102+ }
103+ },
104+ "prisme_b8_vs_b0": {
105+ "per_seed": [
106+ {
107+ "diff": 0.013,
108+ "ci_low": -0.008,
109+ "ci_high": 0.035,
110+ "n": 1000,
111+ "p_two_sided": 0.2546,
112+ "run_id": "p4v2_prisme_seed01_20261030001749"
113+ },
114+ {
115+ "diff": 0.012,
116+ "ci_low": -0.011,
117+ "ci_high": 0.035,
118+ "n": 1000,
119+ "p_two_sided": 0.327,
120+ "run_id": "p4v2_prisme_seed02_20261030001749"
121+ },
122+ {
123+ "diff": 0.023,
124+ "ci_low": 0.001,
125+ "ci_high": 0.045,
126+ "n": 1000,
127+ "p_two_sided": 0.0492,
128+ "run_id": "p4v2_prisme_seed03_20261030001749"
129+ },
130+ {
131+ "diff": 0.013,
132+ "ci_low": -0.009,
133+ "ci_high": 0.036,
134+ "n": 1000,
135+ "p_two_sided": 0.2778,
136+ "run_id": "p4v2_prisme_seed04_20261030001749"
137+ },
138+ {
139+ "diff": 0.014,
140+ "ci_low": -0.008,
141+ "ci_high": 0.036,
142+ "n": 1000,
143+ "p_two_sided": 0.2264,
144+ "run_id": "p4v2_prisme_seed05_20261030001839"
145+ }
146+ ],
147+ "seeds": {
148+ "mean": 0.015,
149+ "sd": 0.004527692569068708,
150+ "ci_low": 0.009378127142052675,
151+ "ci_high": 0.020621872857947324,
152+ "n": 5
153+ }
154+ },
155+ "gr_b1_vs_b0": {
156+ "per_seed": [
157+ {
158+ "diff": 0.025,
159+ "ci_low": 0.01,
160+ "ci_high": 0.041,
161+ "n": 1000,
162+ "p_two_sided": 0.0008,
163+ "run_id": "p4v2_gr_seed01_20261030001749"
164+ },
165+ {
166+ "diff": 0.018,
167+ "ci_low": 0.002,
168+ "ci_high": 0.034,
169+ "n": 1000,
170+ "p_two_sided": 0.0286,
171+ "run_id": "p4v2_gr_seed02_20261030001749"
172+ },
173+ {
174+ "diff": 0.021,
175+ "ci_low": 0.005,
176+ "ci_high": 0.037,
177+ "n": 1000,
178+ "p_two_sided": 0.011,
179+ "run_id": "p4v2_gr_seed03_20261030001749"
180+ },
181+ {
182+ "diff": 0.024,
183+ "ci_low": 0.008,
184+ "ci_high": 0.04,
185+ "n": 1000,
186+ "p_two_sided": 0.0036,
187+ "run_id": "p4v2_gr_seed04_20261030001749"
188+ },
189+ {
190+ "diff": 0.019,
191+ "ci_low": 0.004,
192+ "ci_high": 0.035,
193+ "n": 1000,
194+ "p_two_sided": 0.0178,
195+ "run_id": "p4v2_gr_seed05_20261030001839"
196+ }
197+ ],
198+ "seeds": {
199+ "mean": 0.0214,
200+ "sd": 0.003049590136395382,
201+ "ci_low": 0.017613433718360546,
202+ "ci_high": 0.025186566281639452,
203+ "n": 5
204+ }
205+ },
206+ "gr_b2_vs_b0": {
207+ "per_seed": [
208+ {
209+ "diff": 0.018,
210+ "ci_low": -0.001,
211+ "ci_high": 0.037,
212+ "n": 1000,
213+ "p_two_sided": 0.0676,
214+ "run_id": "p4v2_gr_seed01_20261030001749"
215+ },
216+ {
217+ "diff": 0.016,
218+ "ci_low": -0.002,
219+ "ci_high": 0.035,
220+ "n": 1000,
221+ "p_two_sided": 0.1028,
222+ "run_id": "p4v2_gr_seed02_20261030001749"
223+ },
224+ {
225+ "diff": 0.019,
226+ "ci_low": -0.001,
227+ "ci_high": 0.039,
228+ "n": 1000,
229+ "p_two_sided": 0.0656,
230+ "run_id": "p4v2_gr_seed03_20261030001749"
231+ },
232+ {
233+ "diff": 0.025,
234+ "ci_low": 0.005,
235+ "ci_high": 0.045,
236+ "n": 1000,
237+ "p_two_sided": 0.0126,
238+ "run_id": "p4v2_gr_seed04_20261030001749"
239+ },
240+ {
241+ "diff": 0.015,
242+ "ci_low": -0.003,
243+ "ci_high": 0.034,
244+ "n": 1000,
245+ "p_two_sided": 0.1178,
246+ "run_id": "p4v2_gr_seed05_20261030001839"
247+ }
248+ ],
249+ "seeds": {
250+ "mean": 0.018600000000000002,
251+ "sd": 0.00391152144312159,
252+ "ci_low": 0.013743204645873891,
253+ "ci_high": 0.023456795354126113,
254+ "n": 5
255+ }
256+ },
257+ "gr_b8_vs_b0": {
258+ "per_seed": [
259+ {
260+ "diff": 0.024,
261+ "ci_low": 0.002,
262+ "ci_high": 0.046,
263+ "n": 1000,
264+ "p_two_sided": 0.0338,
265+ "run_id": "p4v2_gr_seed01_20261030001749"
266+ },
267+ {
268+ "diff": 0.025,
269+ "ci_low": 0.003,
270+ "ci_high": 0.047,
271+ "n": 1000,
272+ "p_two_sided": 0.0298,
273+ "run_id": "p4v2_gr_seed02_20261030001749"
274+ },
275+ {
276+ "diff": 0.021,
277+ "ci_low": -0.002,
278+ "ci_high": 0.044,
279+ "n": 1000,
280+ "p_two_sided": 0.0818,
281+ "run_id": "p4v2_gr_seed03_20261030001749"
282+ },
283+ {
284+ "diff": 0.039,
285+ "ci_low": 0.017,
286+ "ci_high": 0.062,
287+ "n": 1000,
288+ "p_two_sided": 0.0014,
289+ "run_id": "p4v2_gr_seed04_20261030001749"
290+ },
291+ {
292+ "diff": 0.008,
293+ "ci_low": -0.014,
294+ "ci_high": 0.03,
295+ "n": 1000,
296+ "p_two_sided": 0.5004,
297+ "run_id": "p4v2_gr_seed05_20261030001839"
298+ }
299+ ],
300+ "seeds": {
301+ "mean": 0.023400000000000004,
302+ "sd": 0.011058933040759403,
303+ "ci_low": 0.009668520984742975,
304+ "ci_high": 0.037131479015257034,
305+ "n": 5
306+ }
307+ },
308+ "prisme_minus_gr_b1": {
309+ "per_seed": [
310+ {
311+ "diff": -0.004,
312+ "ci_low": -0.017,
313+ "ci_high": 0.009,
314+ "n": 1000,
315+ "p_two_sided": 0.6038,
316+ "run_ids": [
317+ "p4v2_prisme_seed01_20261030001749",
318+ "p4v2_gr_seed01_20261030001749"
319+ ]
320+ },
321+ {
322+ "diff": -0.004,
323+ "ci_low": -0.017,
324+ "ci_high": 0.009,
325+ "n": 1000,
326+ "p_two_sided": 0.603,
327+ "run_ids": [
328+ "p4v2_prisme_seed02_20261030001749",
329+ "p4v2_gr_seed02_20261030001749"
330+ ]
331+ },
332+ {
333+ "diff": -0.007,
334+ "ci_low": -0.021,
335+ "ci_high": 0.006,
336+ "n": 1000,
337+ "p_two_sided": 0.3406,
338+ "run_ids": [
339+ "p4v2_prisme_seed03_20261030001749",
340+ "p4v2_gr_seed03_20261030001749"
341+ ]
342+ },
343+ {
344+ "diff": -0.009,
345+ "ci_low": -0.022,
346+ "ci_high": 0.004,
347+ "n": 1000,
348+ "p_two_sided": 0.2072,
349+ "run_ids": [
350+ "p4v2_prisme_seed04_20261030001749",
351+ "p4v2_gr_seed04_20261030001749"
352+ ]
353+ },
354+ {
355+ "diff": -0.004,
356+ "ci_low": -0.017,
357+ "ci_high": 0.008,
358+ "n": 1000,
359+ "p_two_sided": 0.5746,
360+ "run_ids": [
361+ "p4v2_prisme_seed05_20261030001839",
362+ "p4v2_gr_seed05_20261030001839"
363+ ]
364+ }
365+ ],
366+ "seeds": {
367+ "mean": -0.0056,
368+ "sd": 0.0023021728866442675,
369+ "ci_low": -0.008458525190987023,
370+ "ci_high": -0.002741474809012978,
371+ "n": 5
372+ }
373+ },
374+ "prisme_minus_gr_b2": {
375+ "per_seed": [
376+ {
377+ "diff": 0.005,
378+ "ci_low": -0.01,
379+ "ci_high": 0.02,
380+ "n": 1000,
381+ "p_two_sided": 0.5636,
382+ "run_ids": [
383+ "p4v2_prisme_seed01_20261030001749",
384+ "p4v2_gr_seed01_20261030001749"
385+ ]
386+ },
387+ {
388+ "diff": 0.0,
389+ "ci_low": -0.015,
390+ "ci_high": 0.015,
391+ "n": 1000,
392+ "p_two_sided": 1.043,
393+ "run_ids": [
394+ "p4v2_prisme_seed02_20261030001749",
395+ "p4v2_gr_seed02_20261030001749"
396+ ]
397+ },
398+ {
399+ "diff": 0.001,
400+ "ci_low": -0.015,
401+ "ci_high": 0.018,
402+ "n": 1000,
403+ "p_two_sided": 0.955,
404+ "run_ids": [
405+ "p4v2_prisme_seed03_20261030001749",
406+ "p4v2_gr_seed03_20261030001749"
407+ ]
408+ },
409+ {
410+ "diff": -0.007,
411+ "ci_low": -0.022,
412+ "ci_high": 0.008,
413+ "n": 1000,
414+ "p_two_sided": 0.3848,
415+ "run_ids": [
416+ "p4v2_prisme_seed04_20261030001749",
417+ "p4v2_gr_seed04_20261030001749"
418+ ]
419+ },
420+ {
421+ "diff": 0.004,
422+ "ci_low": -0.011,
423+ "ci_high": 0.019,
424+ "n": 1000,
425+ "p_two_sided": 0.6398,
426+ "run_ids": [
427+ "p4v2_prisme_seed05_20261030001839",
428+ "p4v2_gr_seed05_20261030001839"
429+ ]
430+ }
431+ ],
432+ "seeds": {
433+ "mean": 0.0006000000000000001,
434+ "sd": 0.004722287581247038,
435+ "ci_low": -0.00526349447879918,
436+ "ci_high": 0.006463494478799179,
437+ "n": 5
438+ }
439+ },
440+ "prisme_minus_gr_b8": {
441+ "per_seed": [
442+ {
443+ "diff": -0.011,
444+ "ci_low": -0.029,
445+ "ci_high": 0.007,
446+ "n": 1000,
447+ "p_two_sided": 0.264,
448+ "run_ids": [
449+ "p4v2_prisme_seed01_20261030001749",
450+ "p4v2_gr_seed01_20261030001749"
451+ ]
452+ },
453+ {
454+ "diff": -0.013,
455+ "ci_low": -0.032,
456+ "ci_high": 0.005,
457+ "n": 1000,
458+ "p_two_sided": 0.1828,
459+ "run_ids": [
460+ "p4v2_prisme_seed02_20261030001749",
461+ "p4v2_gr_seed02_20261030001749"
462+ ]
463+ },
464+ {
465+ "diff": 0.002,
466+ "ci_low": -0.014,
467+ "ci_high": 0.018,
468+ "n": 1000,
469+ "p_two_sided": 0.8416,
470+ "run_ids": [
471+ "p4v2_prisme_seed03_20261030001749",
472+ "p4v2_gr_seed03_20261030001749"
473+ ]
474+ },
475+ {
476+ "diff": -0.026,
477+ "ci_low": -0.044,
478+ "ci_high": -0.008,
479+ "n": 1000,
480+ "p_two_sided": 0.0044,
481+ "run_ids": [
482+ "p4v2_prisme_seed04_20261030001749",
483+ "p4v2_gr_seed04_20261030001749"
484+ ]
485+ },
486+ {
487+ "diff": 0.006,
488+ "ci_low": -0.012,
489+ "ci_high": 0.025,
490+ "n": 1000,
491+ "p_two_sided": 0.5602,
492+ "run_ids": [
493+ "p4v2_prisme_seed05_20261030001839",
494+ "p4v2_gr_seed05_20261030001839"
495+ ]
496+ }
497+ ],
498+ "seeds": {
499+ "mean": -0.008400000000000001,
500+ "sd": 0.01277888884058391,
501+ "ci_low": -0.02426708621040088,
502+ "ci_high": 0.0074670862104008785,
503+ "n": 5
504+ }
505+ }
506+}
\ No newline at end of file
added results/raw/routing_v1rank.json +290 −0
@@ -0,0 +1,290 @@
1+{
2+ "config": "configs/experiments/routing_v1rank.yaml",
3+ "checkpoint": "experiments/runs/controller_v1rank_seed01_20261030005431/checkpoint",
4+ "checkpoint_sha256": "64570e628212ba370db86265ffaa23f655bfee6e58120829c5bbe5881c7aadd9",
5+ "split": "dev_iid",
6+ "code_commit": "512b98066b4c794b04c46f9b43023c6135f1adc1",
7+ "device": "mps",
8+ "seconds": 84.3,
9+ "records": [
10+ {
11+ "policy": "budget0",
12+ "loss": 0.43343788690653534,
13+ "edits_per_token": 0.0,
14+ "extra_ops_ratio": 0.0,
15+ "n_tokens": 3417
16+ },
17+ {
18+ "policy": "random_forced",
19+ "budget": 1,
20+ "loss": 0.43049107206331766,
21+ "edits_per_token": 1.0,
22+ "extra_ops_ratio": 0.33739269034430613,
23+ "n_tokens": 3417
24+ },
25+ {
26+ "policy": "uniform",
27+ "budget": 1,
28+ "loss": 0.430082754325476,
29+ "edits_per_token": 1.0,
30+ "extra_ops_ratio": 0.2911194881431394,
31+ "n_tokens": 3417
32+ },
33+ {
34+ "policy": "fixed_pair",
35+ "budget": 1,
36+ "loss": 0.4306084942950259,
37+ "edits_per_token": 1.0,
38+ "extra_ops_ratio": 0.38326141294904154,
39+ "n_tokens": 3417
40+ },
41+ {
42+ "policy": "random_forced",
43+ "budget": 2,
44+ "loss": 0.43129151572879726,
45+ "edits_per_token": 2.0,
46+ "extra_ops_ratio": 0.6752437784687831,
47+ "n_tokens": 3417
48+ },
49+ {
50+ "policy": "uniform",
51+ "budget": 2,
52+ "loss": 0.43061076446970825,
53+ "edits_per_token": 2.0,
54+ "extra_ops_ratio": 0.5822389762862789,
55+ "n_tokens": 3417
56+ },
57+ {
58+ "policy": "fixed_pair",
59+ "budget": 2,
60+ "loss": 0.43226337237493456,
61+ "edits_per_token": 2.0,
62+ "extra_ops_ratio": 0.7665228258980831,
63+ "n_tokens": 3417
64+ },
65+ {
66+ "policy": "random_forced",
67+ "budget": 4,
68+ "loss": 0.4350665061488499,
69+ "edits_per_token": 4.0,
70+ "extra_ops_ratio": 1.3500830851558498,
71+ "n_tokens": 3417
72+ },
73+ {
74+ "policy": "uniform",
75+ "budget": 4,
76+ "loss": 0.43226772879940184,
77+ "edits_per_token": 4.0,
78+ "extra_ops_ratio": 1.1644779525725577,
79+ "n_tokens": 3417
80+ },
81+ {
82+ "policy": "fixed_pair",
83+ "budget": 4,
84+ "loss": 0.43608775067685257,
85+ "edits_per_token": 4.0,
86+ "extra_ops_ratio": 1.5330456517961661,
87+ "n_tokens": 3417
88+ },
89+ {
90+ "policy": "random_forced",
91+ "budget": 8,
92+ "loss": 0.44534724575753865,
93+ "edits_per_token": 8.0,
94+ "extra_ops_ratio": 2.6992224028210283,
95+ "n_tokens": 3417
96+ },
97+ {
98+ "policy": "uniform",
99+ "budget": 8,
100+ "loss": 0.4360636224654602,
101+ "edits_per_token": 8.0,
102+ "extra_ops_ratio": 2.3289559051451154,
103+ "n_tokens": 3417
104+ },
105+ {
106+ "policy": "fixed_pair",
107+ "budget": 8,
108+ "loss": 0.4454028259636821,
109+ "edits_per_token": 8.0,
110+ "extra_ops_ratio": 3.0660913035923323,
111+ "n_tokens": 3417
112+ },
113+ {
114+ "policy": "entropy",
115+ "threshold": 0.1,
116+ "budget": 8,
117+ "loss": 0.44345039856653656,
118+ "edits_per_token": 2.893473807433421,
119+ "extra_ops_ratio": 0.9313065950585138,
120+ "n_tokens": 3417
121+ },
122+ {
123+ "policy": "entropy",
124+ "threshold": 0.5,
125+ "budget": 8,
126+ "loss": 0.44244280132174246,
127+ "edits_per_token": 2.097453906935909,
128+ "extra_ops_ratio": 0.6750847836283169,
129+ "n_tokens": 3417
130+ },
131+ {
132+ "policy": "entropy",
133+ "threshold": 1.0,
134+ "budget": 8,
135+ "loss": 0.4375149400603328,
136+ "edits_per_token": 1.1785191688615744,
137+ "extra_ops_ratio": 0.3794397201180842,
138+ "n_tokens": 3417
139+ },
140+ {
141+ "policy": "entropy",
142+ "threshold": 2.0,
143+ "budget": 8,
144+ "loss": 0.43457157094819804,
145+ "edits_per_token": 0.20749195200468246,
146+ "extra_ops_ratio": 0.06636437945523596,
147+ "n_tokens": 3417
148+ },
149+ {
150+ "policy": "norm",
151+ "threshold": 0.02,
152+ "budget": 8,
153+ "loss": 0.4431566372928056,
154+ "edits_per_token": 7.994146912496342,
155+ "extra_ops_ratio": 2.576118717076383,
156+ "n_tokens": 3417
157+ },
158+ {
159+ "policy": "norm",
160+ "threshold": 0.05,
161+ "budget": 8,
162+ "loss": 0.4419520246658124,
163+ "edits_per_token": 7.054141059408838,
164+ "extra_ops_ratio": 2.2736114108984604,
165+ "n_tokens": 3417
166+ },
167+ {
168+ "policy": "norm",
169+ "threshold": 0.1,
170+ "budget": 8,
171+ "loss": 0.4397261778374042,
172+ "edits_per_token": 4.717588527948493,
173+ "extra_ops_ratio": 1.5199677263884506,
174+ "n_tokens": 3417
175+ },
176+ {
177+ "policy": "norm",
178+ "threshold": 0.2,
179+ "budget": 8,
180+ "loss": 0.430901125784258,
181+ "edits_per_token": 1.0491659350307287,
182+ "extra_ops_ratio": 0.33984092665514054,
183+ "n_tokens": 3417
184+ },
185+ {
186+ "policy": "controller_locked",
187+ "lam": 0.3,
188+ "kap": 0.5,
189+ "budget": 8,
190+ "loss": 0.43332059026845965,
191+ "edits_per_token": 0.06613988879133743,
192+ "extra_ops_ratio": 0.01925461050499103,
193+ "n_tokens": 3417
194+ },
195+ {
196+ "policy": "controller",
197+ "lam": 0.0,
198+ "kap": 0.5,
199+ "budget": 8,
200+ "loss": 0.4392198074760582,
201+ "edits_per_token": 1.4641498390400935,
202+ "extra_ops_ratio": 0.48109084149707576,
203+ "n_tokens": 3417
204+ },
205+ {
206+ "policy": "controller",
207+ "lam": 0.01,
208+ "kap": 0.5,
209+ "budget": 8,
210+ "loss": 0.43936958569757484,
211+ "edits_per_token": 1.3075797483172373,
212+ "extra_ops_ratio": 0.42456216117182355,
213+ "n_tokens": 3417
214+ },
215+ {
216+ "policy": "controller",
217+ "lam": 0.03,
218+ "kap": 0.5,
219+ "budget": 8,
220+ "loss": 0.438084998115448,
221+ "edits_per_token": 1.0398009950248757,
222+ "extra_ops_ratio": 0.3312630400798598,
223+ "n_tokens": 3417
224+ },
225+ {
226+ "policy": "controller",
227+ "lam": 0.1,
228+ "kap": 0.5,
229+ "budget": 8,
230+ "loss": 0.4354970293991183,
231+ "edits_per_token": 0.4433713784021071,
232+ "extra_ops_ratio": 0.13166276062654758,
233+ "n_tokens": 3417
234+ },
235+ {
236+ "policy": "controller",
237+ "lam": 0.3,
238+ "kap": 0.5,
239+ "budget": 8,
240+ "loss": 0.43332059026845965,
241+ "edits_per_token": 0.06613988879133743,
242+ "extra_ops_ratio": 0.01925461050499103,
243+ "n_tokens": 3417
244+ },
245+ {
246+ "policy": "oracle1_greedy",
247+ "budget": 1,
248+ "diagnostic": true,
249+ "loss": 0.4025459464804867,
250+ "edits_per_token": 0.5237366003062787,
251+ "extra_ops_ratio": 0.19226168855956283,
252+ "n_tokens": 653
253+ },
254+ {
255+ "policy": "oracle1_greedy",
256+ "budget": 2,
257+ "diagnostic": true,
258+ "loss": 0.390496985653093,
259+ "edits_per_token": 0.9509954058192955,
260+ "extra_ops_ratio": 0.34655941646277993,
261+ "n_tokens": 653
262+ },
263+ {
264+ "policy": "oracle1_greedy",
265+ "budget": 4,
266+ "diagnostic": true,
267+ "loss": 0.37995706181431255,
268+ "edits_per_token": 1.5313935681470139,
269+ "extra_ops_ratio": 0.552776494398599,
270+ "n_tokens": 653
271+ },
272+ {
273+ "policy": "oracle1_greedy",
274+ "budget": 8,
275+ "diagnostic": true,
276+ "loss": 0.3665898426751083,
277+ "edits_per_token": 2.2664624808575806,
278+ "extra_ops_ratio": 0.8196839220125861,
279+ "n_tokens": 653
280+ },
281+ {
282+ "policy": "oracle_two_step",
283+ "diagnostic": true,
284+ "one_step_gain": 0.023339285247627347,
285+ "two_step_gain": 0.03395615775009681,
286+ "greedy_then_best_gain": 0.03374234835306803,
287+ "n_tokens": 174
288+ }
289+ ]
290+}
\ No newline at end of file
added results/raw/routing_v2s1.json +290 −0
@@ -0,0 +1,290 @@
1+{
2+ "config": "configs/experiments/routing_v2s1.yaml",
3+ "checkpoint": "experiments/runs/controller_v2s1_seed01_20261030005758/checkpoint",
4+ "checkpoint_sha256": "61a4dda3056e6047dd0671e53f2bb195aaa6d90f3fc112cc93fcf80d84ba4198",
5+ "split": "dev_iid",
6+ "code_commit": "unknown",
7+ "device": "mps",
8+ "seconds": 50.9,
9+ "records": [
10+ {
11+ "policy": "budget0",
12+ "loss": 0.4965549189338985,
13+ "edits_per_token": 0.0,
14+ "extra_ops_ratio": 0.0,
15+ "n_tokens": 3417
16+ },
17+ {
18+ "policy": "random_forced",
19+ "budget": 1,
20+ "loss": 0.48360088745822744,
21+ "edits_per_token": 1.0,
22+ "extra_ops_ratio": 0.12365152489067972,
23+ "n_tokens": 3417
24+ },
25+ {
26+ "policy": "uniform",
27+ "budget": 1,
28+ "loss": 0.4858294641340131,
29+ "edits_per_token": 1.0,
30+ "extra_ops_ratio": 0.10679070643245528,
31+ "n_tokens": 3417
32+ },
33+ {
34+ "policy": "fixed_pair",
35+ "budget": 1,
36+ "loss": 0.48118645258036347,
37+ "edits_per_token": 1.0,
38+ "extra_ops_ratio": 0.14036496061660547,
39+ "n_tokens": 3417
40+ },
41+ {
42+ "policy": "random_forced",
43+ "budget": 2,
44+ "loss": 0.4784298638066668,
45+ "edits_per_token": 2.0,
46+ "extra_ops_ratio": 0.24747007926514458,
47+ "n_tokens": 3417
48+ },
49+ {
50+ "policy": "uniform",
51+ "budget": 2,
52+ "loss": 0.48122175722956695,
53+ "edits_per_token": 2.0,
54+ "extra_ops_ratio": 0.21358141286491056,
55+ "n_tokens": 3417
56+ },
57+ {
58+ "policy": "fixed_pair",
59+ "budget": 2,
60+ "loss": 0.47615878792021193,
61+ "edits_per_token": 2.0,
62+ "extra_ops_ratio": 0.28072992123321094,
63+ "n_tokens": 3417
64+ },
65+ {
66+ "policy": "random_forced",
67+ "budget": 4,
68+ "loss": 0.4737940016007751,
69+ "edits_per_token": 4.0,
70+ "extra_ops_ratio": 0.4947927651923059,
71+ "n_tokens": 3417
72+ },
73+ {
74+ "policy": "uniform",
75+ "budget": 4,
76+ "loss": 0.4761925692582012,
77+ "edits_per_token": 4.0,
78+ "extra_ops_ratio": 0.4271628257298211,
79+ "n_tokens": 3417
80+ },
81+ {
82+ "policy": "fixed_pair",
83+ "budget": 4,
84+ "loss": 0.47439178989266384,
85+ "edits_per_token": 4.0,
86+ "extra_ops_ratio": 0.5614598424664219,
87+ "n_tokens": 3417
88+ },
89+ {
90+ "policy": "random_forced",
91+ "budget": 8,
92+ "loss": 0.479933912027709,
93+ "edits_per_token": 8.0,
94+ "extra_ops_ratio": 0.9892416248762507,
95+ "n_tokens": 3417
96+ },
97+ {
98+ "policy": "uniform",
99+ "budget": 8,
100+ "loss": 0.4744012490468727,
101+ "edits_per_token": 8.0,
102+ "extra_ops_ratio": 0.8543256514596422,
103+ "n_tokens": 3417
104+ },
105+ {
106+ "policy": "fixed_pair",
107+ "budget": 8,
108+ "loss": 0.48843245580247313,
109+ "edits_per_token": 8.0,
110+ "extra_ops_ratio": 1.1229196849328438,
111+ "n_tokens": 3417
112+ },
113+ {
114+ "policy": "entropy",
115+ "threshold": 0.1,
116+ "budget": 8,
117+ "loss": 0.47797408758425247,
118+ "edits_per_token": 3.0755048287971904,
119+ "extra_ops_ratio": 0.3631002193380595,
120+ "n_tokens": 3417
121+ },
122+ {
123+ "policy": "entropy",
124+ "threshold": 0.5,
125+ "budget": 8,
126+ "loss": 0.4820516032017146,
127+ "edits_per_token": 2.4451273046532047,
128+ "extra_ops_ratio": 0.28835356525067246,
129+ "n_tokens": 3417
130+ },
131+ {
132+ "policy": "entropy",
133+ "threshold": 1.0,
134+ "budget": 8,
135+ "loss": 0.4866514847855628,
136+ "edits_per_token": 1.5993561603745976,
137+ "extra_ops_ratio": 0.1888264363224408,
138+ "n_tokens": 3417
139+ },
140+ {
141+ "policy": "entropy",
142+ "threshold": 2.0,
143+ "budget": 8,
144+ "loss": 0.4954935511476574,
145+ "edits_per_token": 0.4890254609306409,
146+ "extra_ops_ratio": 0.05748009566574164,
147+ "n_tokens": 3417
148+ },
149+ {
150+ "policy": "norm",
151+ "threshold": 0.02,
152+ "budget": 8,
153+ "loss": 0.47761174995437994,
154+ "edits_per_token": 8.0,
155+ "extra_ops_ratio": 0.9450654991893417,
156+ "n_tokens": 3417
157+ },
158+ {
159+ "policy": "norm",
160+ "threshold": 0.05,
161+ "budget": 8,
162+ "loss": 0.4775530145570902,
163+ "edits_per_token": 7.89844893181153,
164+ "extra_ops_ratio": 0.9331399762982823,
165+ "n_tokens": 3417
166+ },
167+ {
168+ "policy": "norm",
169+ "threshold": 0.1,
170+ "budget": 8,
171+ "loss": 0.4774494416747904,
172+ "edits_per_token": 5.593210418495756,
173+ "extra_ops_ratio": 0.6607864040500908,
174+ "n_tokens": 3417
175+ },
176+ {
177+ "policy": "norm",
178+ "threshold": 0.2,
179+ "budget": 8,
180+ "loss": 0.484189420057592,
181+ "edits_per_token": 1.0523851331577407,
182+ "extra_ops_ratio": 0.12498143781930479,
183+ "n_tokens": 3417
184+ },
185+ {
186+ "policy": "controller_locked",
187+ "lam": 0.3,
188+ "kap": 0.5,
189+ "budget": 8,
190+ "loss": 0.4950156743532738,
191+ "edits_per_token": 0.18027509511267192,
192+ "extra_ops_ratio": 0.019261529109336355,
193+ "n_tokens": 3417
194+ },
195+ {
196+ "policy": "controller",
197+ "lam": 0.0,
198+ "kap": 0.5,
199+ "budget": 8,
200+ "loss": 0.4933141067009207,
201+ "edits_per_token": 2.0714076675446296,
202+ "extra_ops_ratio": 0.24507358529814022,
203+ "n_tokens": 3417
204+ },
205+ {
206+ "policy": "controller",
207+ "lam": 0.01,
208+ "kap": 0.5,
209+ "budget": 8,
210+ "loss": 0.4924799355631653,
211+ "edits_per_token": 1.9534679543459175,
212+ "extra_ops_ratio": 0.23033672708986375,
213+ "n_tokens": 3417
214+ },
215+ {
216+ "policy": "controller",
217+ "lam": 0.03,
218+ "kap": 0.5,
219+ "budget": 8,
220+ "loss": 0.49286034918682664,
221+ "edits_per_token": 1.748902546093064,
222+ "extra_ops_ratio": 0.2050815436586777,
223+ "n_tokens": 3417
224+ },
225+ {
226+ "policy": "controller",
227+ "lam": 0.1,
228+ "kap": 0.5,
229+ "budget": 8,
230+ "loss": 0.4940115225604657,
231+ "edits_per_token": 1.0570676031606672,
232+ "extra_ops_ratio": 0.12090272048010196,
233+ "n_tokens": 3417
234+ },
235+ {
236+ "policy": "controller",
237+ "lam": 0.3,
238+ "kap": 0.5,
239+ "budget": 8,
240+ "loss": 0.4950156743532738,
241+ "edits_per_token": 0.18027509511267192,
242+ "extra_ops_ratio": 0.019261529109336355,
243+ "n_tokens": 3417
244+ },
245+ {
246+ "policy": "oracle1_greedy",
247+ "budget": 1,
248+ "diagnostic": true,
249+ "loss": 0.48106685130190885,
250+ "edits_per_token": 0.4333843797856049,
251+ "extra_ops_ratio": 0.05939235219911631,
252+ "n_tokens": 653
253+ },
254+ {
255+ "policy": "oracle1_greedy",
256+ "budget": 2,
257+ "diagnostic": true,
258+ "loss": 0.4616401316013307,
259+ "edits_per_token": 0.7978560490045942,
260+ "extra_ops_ratio": 0.10911176690281259,
261+ "n_tokens": 653
262+ },
263+ {
264+ "policy": "oracle1_greedy",
265+ "budget": 4,
266+ "diagnostic": true,
267+ "loss": 0.4398064138695803,
268+ "edits_per_token": 1.3905053598774886,
269+ "extra_ops_ratio": 0.18839140069795057,
270+ "n_tokens": 653
271+ },
272+ {
273+ "policy": "oracle1_greedy",
274+ "budget": 8,
275+ "diagnostic": true,
276+ "loss": 0.4197931946875673,
277+ "edits_per_token": 2.1623277182235836,
278+ "extra_ops_ratio": 0.2941060263383991,
279+ "n_tokens": 653
280+ },
281+ {
282+ "policy": "oracle_two_step",
283+ "diagnostic": true,
284+ "one_step_gain": 0.018501395466683924,
285+ "two_step_gain": 0.03580966489068393,
286+ "greedy_then_best_gain": 0.03554030122428105,
287+ "n_tokens": 174
288+ }
289+ ]
290+}
\ No newline at end of file
added scripts/statistics.py +59 −0
@@ -0,0 +1,59 @@
1+#!/usr/bin/env python
2+"""Pre-registered uncertainty analysis: paired item-level bootstrap (10,000 resamples) of exact-match differences
3+within each seed, and t-intervals over seeds. Writes results/analyzed/statistics_v2.json and paper/tables/stats.tex."""
4+import glob, json, sys
5+from pathlib import Path
6+import numpy as np
7+sys.path.insert(0, ".")
8+from evaluation.statistical_analysis import paired_bootstrap, seed_interval
9+
10+def load(pre):
11+ out = []
12+ for run in sorted(glob.glob(f"experiments/runs/{pre}_seed*")):
13+ if json.load(open(Path(run) / "manifest.json")).get("state") != "completed":
14+ continue
15+ recs = {}
16+ for b in (0, 1, 2, 8):
17+ p = Path(run) / f"em_dev_iid_b{b}.json"
18+ if p.exists():
19+ recs[b] = {r["uid"]: r["correct"] for r in json.load(open(p))}
20+ out.append((Path(run).name, recs))
21+ return out
22+
23+res = {}
24+systems = {"prisme": load("p4v2_prisme"), "gr": load("p4v2_gr"), "dense": load("p4v2_dense")}
25+for name, runs in systems.items():
26+ for b in (1, 2, 8):
27+ diffs, boots = [], []
28+ for rid, recs in runs:
29+ if b in recs and 0 in recs:
30+ common = sorted(set(recs[0]) & set(recs[b]))
31+ a = np.array([recs[b][u] for u in common]); c = np.array([recs[0][u] for u in common])
32+ bs = paired_bootstrap(a, c); bs["run_id"] = rid; boots.append(bs); diffs.append(bs["diff"])
33+ if diffs:
34+ res[f"{name}_b{b}_vs_b0"] = {"per_seed": boots, "seeds": seed_interval(diffs)}
35+# PRISME vs GR at the same budget (paired by item, per seed index)
36+for b in (1, 2, 8):
37+ diffs, boots = [], []
38+ for (rp, rec_p), (rg, rec_g) in zip(systems["prisme"], systems["gr"]):
39+ if b in rec_p and b in rec_g:
40+ common = sorted(set(rec_p[b]) & set(rec_g[b]))
41+ a = np.array([rec_p[b][u] for u in common]); c = np.array([rec_g[b][u] for u in common])
42+ bs = paired_bootstrap(a, c); bs["run_ids"] = [rp, rg]; boots.append(bs); diffs.append(bs["diff"])
43+ if diffs:
44+ res[f"prisme_minus_gr_b{b}"] = {"per_seed": boots, "seeds": seed_interval(diffs)}
45+Path("results/analyzed").mkdir(exist_ok=True, parents=True)
46+Path("results/analyzed/statistics_v2.json").write_text(json.dumps(res, indent=1))
47+rows = []
48+for k, v in res.items():
49+ s = v["seeds"]; lo = min(b["ci_low"] for b in v["per_seed"]); hi = max(b["ci_high"] for b in v["per_seed"])
50+ rows.append(f"{k.replace('_', ' ')} & {s['n']} & {100*s['mean']:+.2f} & [{100*s['ci_low']:+.2f}, {100*s['ci_high']:+.2f}] & [{100*lo:+.2f}, {100*hi:+.2f}]\\\\")
51+ print(k, "seeds", s["n"], f"mean {100*s['mean']:+.2f} pp, t-CI [{100*s['ci_low']:+.2f}, {100*s['ci_high']:+.2f}]")
52+Path("paper/tables/stats.tex").write_text(r"""\begin{table}[t]\centering\small
53+\caption{Pre-registered uncertainty analysis of dev-iid exact match (percentage points). ``t-CI'' is the 95\,\%
54+$t$-interval over independent adapter seeds of the per-seed paired difference; ``item CI range'' spans the
55+smallest lower and largest upper bound of the per-seed paired item bootstraps (10,000 resamples, 1,000 items).}
56+\label{tab:stats}
57+\begin{tabular}{lrrll}\toprule
58+Comparison & seeds & mean & t-CI over seeds & item CI range\\\midrule
59+""" + "\n".join(rows) + "\n" + r"\bottomrule\end{tabular}\end{table}" + "\n")
60
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere