WP31: phases 4-7 at tiny scale, baselines, evaluation modules, paper skeleton
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
141 changed files +7,118 −29
added
CITATION.cff
+11 −0
@@ -0,0 +1,11 @@ | ||
| 1 | +cff-version: 1.2.0 | |
| 2 | +message: "If you use this software or paper, please cite it as below." | |
| 3 | +title: "PRISME: Intervention-Supervised Sparse Workspace Editing for Small Language Models (UQO Working Paper 31)" | |
| 4 | +authors: | |
| 5 | + - family-names: Boucher | |
| 6 | + given-names: Simon-Pierre | |
| 7 | + affiliation: "Université du Québec en Outaouais" | |
| 8 | + email: simon-pierre.boucher@uqo.ca | |
| 9 | +date-released: 2026-10-30 | |
| 10 | +version: "1.0" | |
| 11 | +license: MIT | |
modified
FAILURES.md
+5 −0
@@ -5,3 +5,8 @@ | ||
| 5 | 5 | | 2026-10-29 | `test_incremental_equals_full_prefix_forced`: cached decoding ≠ full-prefix forward. | pytest, atol 1e-4 | Root cause: forced-action forward let edits resume after STOP; made STOP absorbing (D4). | |
| 6 | 6 | | 2026-10-29 | rsync exclusion `experiments` also dropped `configs/experiments/`; baseline launch on M3U96b failed twice (`FileNotFoundError`). | `experiments/logs/pretrain_enlarged102m_seed1.log` (first attempts) | anchored pattern `/experiments`. | |
| 7 | 7 | | 2026-10-29 | Stage B bootstrap degeneracy (brief §13.1 confirmed): with U = 0 and lr 1e-3, 700 steps of forced mixed-budget training left val loss identical at every budget (0.4144 / 0.4143 / 0.4143 / 0.4146 / 0.4164 for budgets 0/1/2/4/8); grad-norm 0.01–0.02. Diagnosis: scale mismatch — backbone residual rms 5.64 vs readout difference rms 0.06, so U must grow by ×100 before edits can affect logits. | `experiments/runs/p4_editor_v1_seed01_*/metrics.jsonl` (stopped at step ~750) | D9: fixed non-learned `readout_scale` = measured backbone residual rms at bootstrap (keeps U = 0 init and exact identity; equivalent to a scale-aware learning rate). Variants with nonzero U init (std 0.02) and lr 3e-3 kept running on M2U64 / M4M64c as evidence. | |
| 8 | +| 2026-10-30 | Editor collapse at lr 3e-3 (w = 128 mixed budgets): after 500 steps every budget gives exactly the budget-0 loss (0.4005) and the trainable gradient norm is 0.00 — the adapter learned to make all edits inert (gate/U → 0) instead of useful. | M4M64a `experiments/logs/p4_diag_mixed_w128_lr3e3.log`, run `p4_diag_mixed_w128_lr3e3_seed01_*` | "harmless collapse" is a real failure mode of the bootstrap; lr 1e-3 retained. | | |
| 9 | +| 2026-10-30 | Global-recurrence baseline run crashed at the final exact-match evaluation (`AttributeError: 'GlobalRecurrenceModel' object has no attribute 'W_K'`): the cached generator assumed PRISME caches. Training and the step-3000 checkpoint are intact. | M5M36 `experiments/logs/p4_gr_mixed_w64.log` | added `generate_uncached` fallback (full recomputation) for models without the bank; EM re-run from the checkpoint. | | |
| 10 | +| 2026-10-30 | Joint fine-tuning reference (backbone unfrozen, lr 3e-4, answer-span loss): validation loss rose from 0.4005 to 0.472 at every budget after 500 steps — the backbone degraded under the narrower objective and the large learning rate. | M4M64a `experiments/logs/p4_joint_mixed_w64.log` | Not a usable upper reference as run; a low-lr variant is deferred (not required by the pre-registered comparison). | | |
| 11 | +| 2026-10-30 | v2 tiny backbone (5.6M, 15k planned steps) overfit the 23k-item train split: validation loss (all tokens) 0.246 at step 6k → 0.282 at step 9k. Run stopped at step 9,300; the step-5,000 checkpoint (val 0.249) is used as the frozen v2 backbone. | `experiments/logs/p4_tiny_backbone_v2.log`, run `p4_tiny_backbone_v2_seed01_*` (state: stopped) | 42 epochs of a small synthetic set is too many; schedule not re-run (the checkpoint is adequate as a frozen comparator). | | |
| 12 | +| 2026-10-30 | Killed to free nodes for the 5-seed matrix: the degraded joint-finetune reference (M4M64a) and the v1 dense-writes run on the slow m4mn node (superseded by the v2 matrix `p4v2_dense`). | manifests remain in `experiments/runs/` on the nodes (state: running → killed) | — | | |
modified
PROGRESS.md
+7 −1
@@ -6,5 +6,11 @@ | ||
| 6 | 6 | | 2026-10-29 | 1 literature | done | `literature/papers.bib` (77 verified entries), `comparison_matrix.csv`, `novelty_audit.md` | all 12 mandated refs exist incl. RecurTrace 2609.03379 and ELR 2610.12582; C3 (differential zero-init bypass) judged not novel; C2 novel only in its multi-action+propensity form | PASS | preregistration frozen before any test split is read (see protocol/) | |
| 7 | 7 | | 2026-10-29 | 2 backbone + counts | done | `scripts/count_params.py --all-configs --device meta`; `scripts/estimate_flops.py` | backbone counts match brief exactly (99,502,848 / 446,744,064 / 1,149,339,648); workspace 2,631,173 vs brief 3,088,260 (reconciled, D3); ops table reproduces brief within 1.3 % | PASS | — | |
| 8 | 8 | | 2026-10-29 | 3 invariants | done | `python -m pytest tests -q` → 22 passed | 15 mandated invariants + controller target-freedom + dense ablation + 5 cache-equivalence tests | PASS | — | |
| 9 | −| 2026-10-29 | 4 learnability | running | `training.train_editor --config configs/experiments/p4_tiny_backbone.yaml --device mps` (laptop) | — | — | then editor bootstrap on frozen tiny backbone | | |
| 9 | +| 2026-10-29 | 4 learnability (backbone) | done | `p4_tiny_backbone.yaml` → run `p4_tiny_backbone_seed01_20261029232626` (2.1M params, 6k steps, char-level) | exact match (greedy): dev_iid arith 16.8 %, dev_template 13.7 %, dev_depth 3.0 %, compose 2.4 % (first-600-item subsets, arith-heavy) | PASS (weak backbone, usable comparator) | stronger v2 backbone (5.6M, 15k steps) running | | |
| 10 | +| 2026-10-29 | 4 learnability (overfit) | done | `p4_overfit64_b8.yaml` → run `p4_overfit64_b8_seed01_20261029235111` | 64 items, budget 8, frozen backbone: train loss 0.347 → 0.0024 in 200 steps; dev loss rises 0.73 → 1.36 (expected) | PASS (gradient path + capacity) | — | | |
| 11 | +| 2026-10-29 | 4 learnability (generalisation, v1 backbone) | done | zero-U scaled (laptop), U std 0.2 / 0.5 (M4M64c / M2U64), budget-8-only w64/w128 and mixed w128 lr 3e-3 (M2U64 / M4M64c / M4M64a) | after 2000 steps (U std 0.2): dev-iid loss −1.4 % at budgets 1–2, +1.7 % at budget 8; exact match dev-iid 43.8 → 45.8 % (b2), template/depth unchanged, compose 2.3 → 3.0 %; zero-U + scale alone: −0.6 %; lr 3e-3 w128: collapse to inert edits; global-recurrence adapter (same recipe): loss ≥ backbone at every budget, EM 43.8 → 45.8 % (b8) | PASS (weak, learnable; mechanism effect ≈ 1 %) | decisive 5-seed matrix on v2 backbone | | |
| 12 | +| 2026-10-30 | 4 decisive matrix (v2 backbone, 5 seeds × {PRISME, global recurrence, dense}) | running | `scripts/launch_seed_matrix.sh` → 15 runs on M2U64/M4M64c/M4M64a/M5M36/M4BP48/M4BP36 | — | — | aggregate with `scripts/make_tables.py` | | |
| 13 | +| 2026-10-30 | 5 counterfactual labels + oracle audit (tiny) | done | `labels_p4.yaml` → run `labels_p4_seed01_20261030000543` (80,632 train + 20,040 calib labels; 4,778 + 1,308 exhaustive-audit states); `evaluation.label_analysis` | gains: mean 0.004, 42 % positive, 37 % harmful; variance 86–89 % between states vs 11–14 % within state (action identity); 37 % of states have all 24 actions positive, 29 % all negative; one-step oracle 0.0166 nats/state = 0.0098 (perfect stop, random action) + 0.0067 (action selection); random action 0.0004 | PASS (oracle margin exists; action choice is ≈40 % of it) | controller fit | | |
| 14 | +| 2026-10-30 | 6 controller (tiny) | partial | `controller_p4.yaml` → `controller_p4_seed01_20261030000716` (z features): calib sign acc 49 %, captured fraction −3 %; sweep: best (λ=0.1, κ=0.5) loss −0.1 % with 0.01 edits/token; `controller_p4_hent.yaml` (h + backbone entropy): sign acc 59–60 % at 1–2k steps then overfits (early stopping added) | controller with z-only features = chance; h+entropy features predict the sign above chance but capture ≤ 5 % of the oracle opportunity | OPEN (H3 null so far at tiny scale) | rerun with early stopping; more labels | | |
| 15 | +| 2026-10-30 | 7 inference engine + latency (tiny) | done | `tests/integration/test_cache_equivalence.py` (5 tests pass); `evaluation.latency` on the laptop MPS → `results/raw/latency_p4_*_mps.json` | batch-1 decode: 2.8 ms/token (budget 0) → 5.3 (1 edit) → 19.3 ms (8 edits); dense 8 edits 17.3 ms (not slower than sparse at this scale); controller call ≈ +2.9 ms even with 0 edits | PASS (equivalence); dispatch overhead dominates at tiny scale (brief §13.6 confirmed) | re-measure at 103M | | |
| 10 | 16 | | 2026-10-29 | 8 pilot (Stage A) | running | M3U96a: `pretrain_103m.yaml` (run `pretrain103m_seed01_*`); M3U96b: `pretrain_enlarged_102m.yaml` | 14.8k tok/s, 196.6M tokens planned (~3.7 h each) | — | editor/controller stages after completion | |
modified
baselines/global_recurrence.py
+6 −2
@@ -50,7 +50,11 @@ class GlobalRecurrenceModel(nn.Module): | ||
| 50 | 50 | self.block = RecurrentBlock(w, heads or cfg.n_editor_heads, ffn_w or cfg.ffn_w, cfg.norm_eps) |
| 51 | 51 | self.ctx_proj = nn.Linear(2, w, bias=True) |
| 52 | 52 | self.up = nn.Linear(w, cfg.d_model, bias=False) |
| 53 | − nn.init.zeros_(self.up.weight); nn.init.zeros_(self.ctx_proj.weight); nn.init.zeros_(self.ctx_proj.bias) | |
| 53 | + nn.init.zeros_(self.ctx_proj.weight); nn.init.zeros_(self.ctx_proj.bias) | |
| 54 | + if cfg.readout_zero_init: | |
| 55 | + nn.init.zeros_(self.up.weight) | |
| 56 | + else: | |
| 57 | + nn.init.normal_(self.up.weight, std=cfg.readout_init_std) | |
| 54 | 58 | self.controller = nn.Identity() # placeholder so parameter-group helpers work |
| 55 | 59 | |
| 56 | 60 | def forward(self, tokens, actions=None, return_state=False): |
@@ -72,7 +76,7 @@ class GlobalRecurrenceModel(nn.Module): | ||
| 72 | 76 | xn = self.block(x + self.ctx_proj(ctx), mask) |
| 73 | 77 | x = torch.where(act.unsqueeze(-1), xn, x) # positions that stopped keep their state |
| 74 | 78 | n_edits += act.long() |
| 75 | − delta = self.up(x - x0) | |
| 79 | + delta = self.up(x - x0) * self.cfg.readout_scale | |
| 76 | 80 | edited = n_edits > 0 |
| 77 | 81 | delta = torch.where(edited.unsqueeze(-1), delta, torch.zeros_like(delta)) |
| 78 | 82 | return {"logits": self.backbone.head(H + delta), "H": H, "n_edits": n_edits, "delta_out": delta} |
modified
configs/experiments/controller_p4.yaml
+3 −3
@@ -2,10 +2,10 @@ | ||
| 2 | 2 | name: controller_p4 |
| 3 | 3 | seed: 1 |
| 4 | 4 | mode: prisme |
| 5 | −model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512, | |
| 5 | +model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512, readout_zero_init: false, readout_init_std: 0.2, readout_scale: 5.251, | |
| 6 | 6 | max_seq_len: 320, w: 64, ffn_w: 192, n_slots: 16, n_editor_heads: 2, r_max: 8} |
| 7 | −checkpoint: CHECKPOINT_DIR | |
| 8 | −labels: {train: LABELS_TRAIN, calib: LABELS_CALIB} | |
| 7 | +checkpoint: experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/checkpoint | |
| 8 | +labels: {train: experiments/runs/labels_p4_seed01_20261030000543/labels_train.pt, calib: experiments/runs/labels_p4_seed01_20261030000543/labels_calib.pt} | |
| 9 | 9 | train: {steps: 3000, lr: 1.0e-3, weight_decay: 0.01, alpha: 1.0, beta: 0.5, gamma: 0.1, huber_delta: 0.1, log_every: 100, eval_every: 1000, seq_len: 320} |
| 10 | 10 | lambda_grid: [0.0, 0.01, 0.03, 0.1, 0.3] |
| 11 | 11 | kappa_grid: [0.0, 0.5] |
added
configs/experiments/controller_p4_hent.yaml
+50 −0
@@ -0,0 +1,50 @@ | ||
| 1 | +name: controller_p4_hent | |
| 2 | +seed: 1 | |
| 3 | +mode: prisme | |
| 4 | +model: | |
| 5 | + name: tiny_p4 | |
| 6 | + vocab_size: 128 | |
| 7 | + d_model: 192 | |
| 8 | + n_layers: 4 | |
| 9 | + n_heads: 6 | |
| 10 | + n_kv_heads: 2 | |
| 11 | + ffn_width: 512 | |
| 12 | + readout_zero_init: false | |
| 13 | + readout_init_std: 0.2 | |
| 14 | + readout_scale: 5.251 | |
| 15 | + max_seq_len: 320 | |
| 16 | + w: 64 | |
| 17 | + ffn_w: 192 | |
| 18 | + n_slots: 16 | |
| 19 | + n_editor_heads: 2 | |
| 20 | + r_max: 8 | |
| 21 | +checkpoint: experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/checkpoint | |
| 22 | +labels: | |
| 23 | + train: experiments/runs/labels_p4_seed01_20261030000543/labels_train.pt | |
| 24 | + calib: experiments/runs/labels_p4_seed01_20261030000543/labels_calib.pt | |
| 25 | +train: | |
| 26 | + steps: 3000 | |
| 27 | + lr: 0.001 | |
| 28 | + weight_decay: 0.01 | |
| 29 | + alpha: 1.0 | |
| 30 | + beta: 0.5 | |
| 31 | + gamma: 0.1 | |
| 32 | + huber_delta: 0.1 | |
| 33 | + log_every: 100 | |
| 34 | + eval_every: 1000 | |
| 35 | + seq_len: 320 | |
| 36 | +lambda_grid: | |
| 37 | +- 0.0 | |
| 38 | +- 0.01 | |
| 39 | +- 0.03 | |
| 40 | +- 0.1 | |
| 41 | +- 0.3 | |
| 42 | +kappa_grid: | |
| 43 | +- 0.0 | |
| 44 | +- 0.5 | |
| 45 | +sweep_batches: 10 | |
| 46 | +max_budget: 8 | |
| 47 | +selection_rule: max_loss_reduction_per_ops | |
| 48 | +model_overrides: | |
| 49 | + ctrl_input: h | |
| 50 | + ctrl_use_entropy: true | |
added
configs/experiments/dense_pilot_103m.yaml
+43 −0
@@ -0,0 +1,43 @@ | ||
| 1 | +seed: 1 | |
| 2 | +model: configs/prisme_103m.yaml | |
| 3 | +model_overrides: | |
| 4 | + readout_zero_init: false | |
| 5 | + readout_init_std: 0.2 | |
| 6 | + ctrl_input: h | |
| 7 | + ctrl_use_entropy: true | |
| 8 | +init: | |
| 9 | + backbone_checkpoint: experiments/runs/PRETRAIN103M_RUN/checkpoint | |
| 10 | + freeze_backbone: true | |
| 11 | + auto_readout_scale: true | |
| 12 | +data: | |
| 13 | + kind: tokens | |
| 14 | + train_bin: data/processed/fineweb_edu_300m/train.bin | |
| 15 | + val_bin: data/processed/fineweb_edu_300m/val.bin | |
| 16 | +budgets: | |
| 17 | +- 0 | |
| 18 | +- 1 | |
| 19 | +- 2 | |
| 20 | +- 4 | |
| 21 | +- 8 | |
| 22 | +budget_probs: | |
| 23 | +- 0.2 | |
| 24 | +- 0.2 | |
| 25 | +- 0.2 | |
| 26 | +- 0.2 | |
| 27 | +- 0.2 | |
| 28 | +budget_per: position | |
| 29 | +train: | |
| 30 | + batch_size: 16 | |
| 31 | + seq_len: 512 | |
| 32 | + steps: 3000 | |
| 33 | + lr: 0.001 | |
| 34 | + warmup: 100 | |
| 35 | + weight_decay: 0.1 | |
| 36 | + grad_clip: 1.0 | |
| 37 | + log_every: 20 | |
| 38 | + eval_every: 500 | |
| 39 | + eval_batches: 10 | |
| 40 | + ckpt_every: 1000 | |
| 41 | +final_exact_match: false | |
| 42 | +name: dense_pilot_103m | |
| 43 | +mode: dense | |
added
configs/experiments/editor_pilot_103m.yaml
+43 −0
@@ -0,0 +1,43 @@ | ||
| 1 | +seed: 1 | |
| 2 | +model: configs/prisme_103m.yaml | |
| 3 | +model_overrides: | |
| 4 | + readout_zero_init: false | |
| 5 | + readout_init_std: 0.2 | |
| 6 | + ctrl_input: h | |
| 7 | + ctrl_use_entropy: true | |
| 8 | +init: | |
| 9 | + backbone_checkpoint: experiments/runs/PRETRAIN103M_RUN/checkpoint | |
| 10 | + freeze_backbone: true | |
| 11 | + auto_readout_scale: true | |
| 12 | +data: | |
| 13 | + kind: tokens | |
| 14 | + train_bin: data/processed/fineweb_edu_300m/train.bin | |
| 15 | + val_bin: data/processed/fineweb_edu_300m/val.bin | |
| 16 | +budgets: | |
| 17 | +- 0 | |
| 18 | +- 1 | |
| 19 | +- 2 | |
| 20 | +- 4 | |
| 21 | +- 8 | |
| 22 | +budget_probs: | |
| 23 | +- 0.2 | |
| 24 | +- 0.2 | |
| 25 | +- 0.2 | |
| 26 | +- 0.2 | |
| 27 | +- 0.2 | |
| 28 | +budget_per: position | |
| 29 | +train: | |
| 30 | + batch_size: 16 | |
| 31 | + seq_len: 512 | |
| 32 | + steps: 3000 | |
| 33 | + lr: 0.001 | |
| 34 | + warmup: 100 | |
| 35 | + weight_decay: 0.1 | |
| 36 | + grad_clip: 1.0 | |
| 37 | + log_every: 20 | |
| 38 | + eval_every: 500 | |
| 39 | + eval_batches: 10 | |
| 40 | + ckpt_every: 1000 | |
| 41 | +final_exact_match: false | |
| 42 | +name: editor_pilot_103m | |
| 43 | +mode: prisme | |
added
configs/experiments/eval_p4_dev_gr.yaml
+15 −0
@@ -0,0 +1,15 @@ | ||
| 1 | +# Exact-match evaluation of the global-recurrence baseline (tiny scale) on the dev regimes. | |
| 2 | +out_dir: results/raw | |
| 3 | +data: {kind: synthetic, tokenizer: char, splits_dir: data/splits/synthetic_v1} | |
| 4 | +splits: [dev_iid, dev_template, dev_depth, dev_compose] | |
| 5 | +batch_size: 64 | |
| 6 | +seq_len: 320 | |
| 7 | +max_items: 400 | |
| 8 | +systems: | |
| 9 | + gr_mixed_w64_seed1: | |
| 10 | + mode: global_recurrence | |
| 11 | + baseline_kwargs: {ffn_w: 640} | |
| 12 | + model: from_checkpoint | |
| 13 | + checkpoint: experiments/runs/p4_gr_mixed_w64_seed01_20261030000500/checkpoint | |
| 14 | + controller: false | |
| 15 | + policies: [budget0, forced_b1, forced_b2, forced_b8] | |
added
configs/experiments/gr_pilot_103m.yaml
+45 −0
@@ -0,0 +1,45 @@ | ||
| 1 | +seed: 1 | |
| 2 | +model: configs/prisme_103m.yaml | |
| 3 | +model_overrides: | |
| 4 | + readout_zero_init: false | |
| 5 | + readout_init_std: 0.2 | |
| 6 | + ctrl_input: h | |
| 7 | + ctrl_use_entropy: true | |
| 8 | +init: | |
| 9 | + backbone_checkpoint: experiments/runs/PRETRAIN103M_RUN/checkpoint | |
| 10 | + freeze_backbone: true | |
| 11 | + auto_readout_scale: true | |
| 12 | +data: | |
| 13 | + kind: tokens | |
| 14 | + train_bin: data/processed/fineweb_edu_300m/train.bin | |
| 15 | + val_bin: data/processed/fineweb_edu_300m/val.bin | |
| 16 | +budgets: | |
| 17 | +- 0 | |
| 18 | +- 1 | |
| 19 | +- 2 | |
| 20 | +- 4 | |
| 21 | +- 8 | |
| 22 | +budget_probs: | |
| 23 | +- 0.2 | |
| 24 | +- 0.2 | |
| 25 | +- 0.2 | |
| 26 | +- 0.2 | |
| 27 | +- 0.2 | |
| 28 | +budget_per: position | |
| 29 | +train: | |
| 30 | + batch_size: 16 | |
| 31 | + seq_len: 512 | |
| 32 | + steps: 3000 | |
| 33 | + lr: 0.001 | |
| 34 | + warmup: 100 | |
| 35 | + weight_decay: 0.1 | |
| 36 | + grad_clip: 1.0 | |
| 37 | + log_every: 20 | |
| 38 | + eval_every: 500 | |
| 39 | + eval_batches: 10 | |
| 40 | + ckpt_every: 1000 | |
| 41 | +final_exact_match: false | |
| 42 | +name: gr_pilot_103m | |
| 43 | +mode: global_recurrence | |
| 44 | +baseline_kwargs: | |
| 45 | + ffn_w: 2560 | |
modified
configs/experiments/labels_p4.yaml
+2 −2
@@ -2,9 +2,9 @@ | ||
| 2 | 2 | name: labels_p4 |
| 3 | 3 | seed: 1 |
| 4 | 4 | mode: prisme |
| 5 | −model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512, | |
| 5 | +model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512, readout_zero_init: false, readout_init_std: 0.2, readout_scale: 5.251, | |
| 6 | 6 | max_seq_len: 320, w: 64, ffn_w: 192, n_slots: 16, n_editor_heads: 2, r_max: 8} |
| 7 | −checkpoint: CHECKPOINT_DIR | |
| 7 | +checkpoint: experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/checkpoint | |
| 8 | 8 | use_policy: false |
| 9 | 9 | data: {kind: synthetic, tokenizer: char, splits_dir: data/splits/synthetic_v1, loss_span: answer} |
| 10 | 10 | collect: {batch_size: 32, seq_len: 320, max_prefix: 3, audit_every: 4} |
added
configs/experiments/latency_p4.yaml
+5 −0
@@ -0,0 +1,5 @@ | ||
| 1 | +# Tiny-scale latency microbenchmark (laptop or node): editor checkpoint v1 (U std 0.2), controller untrained -> forced budgets only matter. | |
| 2 | +seed: 1 | |
| 3 | +mode: prisme | |
| 4 | +model: from_checkpoint | |
| 5 | +checkpoint: experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/checkpoint | |
added
configs/experiments/p4_dense_mixed_w64.yaml
+64 −0
@@ -0,0 +1,64 @@ | ||
| 1 | +name: p4_dense_mixed_w64 | |
| 2 | +seed: 1 | |
| 3 | +mode: dense | |
| 4 | +model: | |
| 5 | + name: tiny_p4 | |
| 6 | + vocab_size: 128 | |
| 7 | + d_model: 192 | |
| 8 | + n_layers: 4 | |
| 9 | + n_heads: 6 | |
| 10 | + n_kv_heads: 2 | |
| 11 | + ffn_width: 512 | |
| 12 | + max_seq_len: 320 | |
| 13 | + w: 64 | |
| 14 | + ffn_w: 192 | |
| 15 | + n_slots: 16 | |
| 16 | + n_editor_heads: 2 | |
| 17 | + r_max: 8 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint | |
| 20 | + freeze_backbone: true | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + - dev_template | |
| 30 | + - dev_depth | |
| 31 | + - dev_compose | |
| 32 | +budgets: | |
| 33 | +- 0 | |
| 34 | +- 1 | |
| 35 | +- 2 | |
| 36 | +- 4 | |
| 37 | +- 8 | |
| 38 | +budget_probs: | |
| 39 | +- 0.2 | |
| 40 | +- 0.2 | |
| 41 | +- 0.2 | |
| 42 | +- 0.2 | |
| 43 | +- 0.2 | |
| 44 | +budget_per: position | |
| 45 | +train: | |
| 46 | + batch_size: 32 | |
| 47 | + seq_len: 320 | |
| 48 | + steps: 3000 | |
| 49 | + lr: 0.001 | |
| 50 | + warmup: 100 | |
| 51 | + weight_decay: 0.1 | |
| 52 | + grad_clip: 1.0 | |
| 53 | + log_every: 50 | |
| 54 | + eval_every: 500 | |
| 55 | + eval_batches: 12 | |
| 56 | + ckpt_every: 1500 | |
| 57 | +em_budgets: | |
| 58 | +- 0 | |
| 59 | +- 2 | |
| 60 | +- 8 | |
| 61 | +em_max_items: 400 | |
| 62 | +model_overrides: | |
| 63 | + readout_zero_init: false | |
| 64 | + readout_init_std: 0.2 | |
added
configs/experiments/p4_diag_b8only_w128.yaml
+59 −0
@@ -0,0 +1,59 @@ | ||
| 1 | +name: p4_diag_b8only_w128 | |
| 2 | +seed: 1 | |
| 3 | +mode: prisme | |
| 4 | +model: | |
| 5 | + name: tiny_p4 | |
| 6 | + vocab_size: 128 | |
| 7 | + d_model: 192 | |
| 8 | + n_layers: 4 | |
| 9 | + n_heads: 6 | |
| 10 | + n_kv_heads: 2 | |
| 11 | + ffn_width: 512 | |
| 12 | + max_seq_len: 320 | |
| 13 | + w: 64 | |
| 14 | + ffn_w: 192 | |
| 15 | + n_slots: 16 | |
| 16 | + n_editor_heads: 2 | |
| 17 | + r_max: 8 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint | |
| 20 | + freeze_backbone: true | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + - dev_template | |
| 30 | + - dev_depth | |
| 31 | + - dev_compose | |
| 32 | +budgets: | |
| 33 | +- 8 | |
| 34 | +budget_probs: | |
| 35 | +- 1.0 | |
| 36 | +budget_per: position | |
| 37 | +train: | |
| 38 | + batch_size: 32 | |
| 39 | + seq_len: 320 | |
| 40 | + steps: 3000 | |
| 41 | + lr: 0.001 | |
| 42 | + warmup: 100 | |
| 43 | + weight_decay: 0.1 | |
| 44 | + grad_clip: 1.0 | |
| 45 | + log_every: 50 | |
| 46 | + eval_every: 500 | |
| 47 | + eval_batches: 12 | |
| 48 | + ckpt_every: 1500 | |
| 49 | +em_budgets: | |
| 50 | +- 0 | |
| 51 | +- 2 | |
| 52 | +- 8 | |
| 53 | +em_max_items: 400 | |
| 54 | +model_overrides: | |
| 55 | + readout_zero_init: false | |
| 56 | + readout_init_std: 0.2 | |
| 57 | + w: 128 | |
| 58 | + ffn_w: 384 | |
| 59 | + n_editor_heads: 4 | |
added
configs/experiments/p4_diag_b8only_w64.yaml
+56 −0
@@ -0,0 +1,56 @@ | ||
| 1 | +name: p4_diag_b8only_w64 | |
| 2 | +seed: 1 | |
| 3 | +mode: prisme | |
| 4 | +model: | |
| 5 | + name: tiny_p4 | |
| 6 | + vocab_size: 128 | |
| 7 | + d_model: 192 | |
| 8 | + n_layers: 4 | |
| 9 | + n_heads: 6 | |
| 10 | + n_kv_heads: 2 | |
| 11 | + ffn_width: 512 | |
| 12 | + max_seq_len: 320 | |
| 13 | + w: 64 | |
| 14 | + ffn_w: 192 | |
| 15 | + n_slots: 16 | |
| 16 | + n_editor_heads: 2 | |
| 17 | + r_max: 8 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint | |
| 20 | + freeze_backbone: true | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + - dev_template | |
| 30 | + - dev_depth | |
| 31 | + - dev_compose | |
| 32 | +budgets: | |
| 33 | +- 8 | |
| 34 | +budget_probs: | |
| 35 | +- 1.0 | |
| 36 | +budget_per: position | |
| 37 | +train: | |
| 38 | + batch_size: 32 | |
| 39 | + seq_len: 320 | |
| 40 | + steps: 3000 | |
| 41 | + lr: 0.001 | |
| 42 | + warmup: 100 | |
| 43 | + weight_decay: 0.1 | |
| 44 | + grad_clip: 1.0 | |
| 45 | + log_every: 50 | |
| 46 | + eval_every: 500 | |
| 47 | + eval_batches: 12 | |
| 48 | + ckpt_every: 1500 | |
| 49 | +em_budgets: | |
| 50 | +- 0 | |
| 51 | +- 2 | |
| 52 | +- 8 | |
| 53 | +em_max_items: 400 | |
| 54 | +model_overrides: | |
| 55 | + readout_zero_init: false | |
| 56 | + readout_init_std: 0.2 | |
added
configs/experiments/p4_diag_mixed_w128_lr3e3.yaml
+67 −0
@@ -0,0 +1,67 @@ | ||
| 1 | +name: p4_diag_mixed_w128_lr3e3 | |
| 2 | +seed: 1 | |
| 3 | +mode: prisme | |
| 4 | +model: | |
| 5 | + name: tiny_p4 | |
| 6 | + vocab_size: 128 | |
| 7 | + d_model: 192 | |
| 8 | + n_layers: 4 | |
| 9 | + n_heads: 6 | |
| 10 | + n_kv_heads: 2 | |
| 11 | + ffn_width: 512 | |
| 12 | + max_seq_len: 320 | |
| 13 | + w: 64 | |
| 14 | + ffn_w: 192 | |
| 15 | + n_slots: 16 | |
| 16 | + n_editor_heads: 2 | |
| 17 | + r_max: 8 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint | |
| 20 | + freeze_backbone: true | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + - dev_template | |
| 30 | + - dev_depth | |
| 31 | + - dev_compose | |
| 32 | +budgets: | |
| 33 | +- 0 | |
| 34 | +- 1 | |
| 35 | +- 2 | |
| 36 | +- 4 | |
| 37 | +- 8 | |
| 38 | +budget_probs: | |
| 39 | +- 0.2 | |
| 40 | +- 0.2 | |
| 41 | +- 0.2 | |
| 42 | +- 0.2 | |
| 43 | +- 0.2 | |
| 44 | +budget_per: position | |
| 45 | +train: | |
| 46 | + batch_size: 32 | |
| 47 | + seq_len: 320 | |
| 48 | + steps: 3000 | |
| 49 | + lr: 0.003 | |
| 50 | + warmup: 100 | |
| 51 | + weight_decay: 0.1 | |
| 52 | + grad_clip: 1.0 | |
| 53 | + log_every: 50 | |
| 54 | + eval_every: 500 | |
| 55 | + eval_batches: 12 | |
| 56 | + ckpt_every: 1500 | |
| 57 | +em_budgets: | |
| 58 | +- 0 | |
| 59 | +- 2 | |
| 60 | +- 8 | |
| 61 | +em_max_items: 400 | |
| 62 | +model_overrides: | |
| 63 | + readout_zero_init: false | |
| 64 | + readout_init_std: 0.2 | |
| 65 | + w: 128 | |
| 66 | + ffn_w: 384 | |
| 67 | + n_editor_heads: 4 | |
added
configs/experiments/p4_editor_v1_Ustd02.yaml
+64 −0
@@ -0,0 +1,64 @@ | ||
| 1 | +name: p4_editor_v1_Ustd02 | |
| 2 | +seed: 1 | |
| 3 | +mode: prisme | |
| 4 | +model: | |
| 5 | + name: tiny_p4 | |
| 6 | + vocab_size: 128 | |
| 7 | + d_model: 192 | |
| 8 | + n_layers: 4 | |
| 9 | + n_heads: 6 | |
| 10 | + n_kv_heads: 2 | |
| 11 | + ffn_width: 512 | |
| 12 | + max_seq_len: 320 | |
| 13 | + w: 64 | |
| 14 | + ffn_w: 192 | |
| 15 | + n_slots: 16 | |
| 16 | + n_editor_heads: 2 | |
| 17 | + r_max: 8 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint | |
| 20 | + freeze_backbone: true | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + - dev_template | |
| 30 | + - dev_depth | |
| 31 | + - dev_compose | |
| 32 | +budgets: | |
| 33 | +- 0 | |
| 34 | +- 1 | |
| 35 | +- 2 | |
| 36 | +- 4 | |
| 37 | +- 8 | |
| 38 | +budget_probs: | |
| 39 | +- 0.2 | |
| 40 | +- 0.2 | |
| 41 | +- 0.2 | |
| 42 | +- 0.2 | |
| 43 | +- 0.2 | |
| 44 | +budget_per: position | |
| 45 | +train: | |
| 46 | + batch_size: 32 | |
| 47 | + seq_len: 320 | |
| 48 | + steps: 2000 | |
| 49 | + lr: 0.001 | |
| 50 | + warmup: 100 | |
| 51 | + weight_decay: 0.1 | |
| 52 | + grad_clip: 1.0 | |
| 53 | + log_every: 50 | |
| 54 | + eval_every: 500 | |
| 55 | + eval_batches: 8 | |
| 56 | + ckpt_every: 1500 | |
| 57 | +em_budgets: | |
| 58 | +- 0 | |
| 59 | +- 2 | |
| 60 | +- 8 | |
| 61 | +em_max_items: 400 | |
| 62 | +model_overrides: | |
| 63 | + readout_zero_init: false | |
| 64 | + readout_init_std: 0.2 | |
added
configs/experiments/p4_editor_v1_Ustd05.yaml
+64 −0
@@ -0,0 +1,64 @@ | ||
| 1 | +name: p4_editor_v1_Ustd05 | |
| 2 | +seed: 1 | |
| 3 | +mode: prisme | |
| 4 | +model: | |
| 5 | + name: tiny_p4 | |
| 6 | + vocab_size: 128 | |
| 7 | + d_model: 192 | |
| 8 | + n_layers: 4 | |
| 9 | + n_heads: 6 | |
| 10 | + n_kv_heads: 2 | |
| 11 | + ffn_width: 512 | |
| 12 | + max_seq_len: 320 | |
| 13 | + w: 64 | |
| 14 | + ffn_w: 192 | |
| 15 | + n_slots: 16 | |
| 16 | + n_editor_heads: 2 | |
| 17 | + r_max: 8 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint | |
| 20 | + freeze_backbone: true | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + - dev_template | |
| 30 | + - dev_depth | |
| 31 | + - dev_compose | |
| 32 | +budgets: | |
| 33 | +- 0 | |
| 34 | +- 1 | |
| 35 | +- 2 | |
| 36 | +- 4 | |
| 37 | +- 8 | |
| 38 | +budget_probs: | |
| 39 | +- 0.2 | |
| 40 | +- 0.2 | |
| 41 | +- 0.2 | |
| 42 | +- 0.2 | |
| 43 | +- 0.2 | |
| 44 | +budget_per: position | |
| 45 | +train: | |
| 46 | + batch_size: 32 | |
| 47 | + seq_len: 320 | |
| 48 | + steps: 2000 | |
| 49 | + lr: 0.001 | |
| 50 | + warmup: 100 | |
| 51 | + weight_decay: 0.1 | |
| 52 | + grad_clip: 1.0 | |
| 53 | + log_every: 50 | |
| 54 | + eval_every: 500 | |
| 55 | + eval_batches: 8 | |
| 56 | + ckpt_every: 1500 | |
| 57 | +em_budgets: | |
| 58 | +- 0 | |
| 59 | +- 2 | |
| 60 | +- 8 | |
| 61 | +em_max_items: 400 | |
| 62 | +model_overrides: | |
| 63 | + readout_zero_init: false | |
| 64 | + readout_init_std: 0.5 | |
added
configs/experiments/p4_gr_mixed_w64.yaml
+66 −0
@@ -0,0 +1,66 @@ | ||
| 1 | +name: p4_gr_mixed_w64 | |
| 2 | +seed: 1 | |
| 3 | +mode: global_recurrence | |
| 4 | +model: | |
| 5 | + name: tiny_p4 | |
| 6 | + vocab_size: 128 | |
| 7 | + d_model: 192 | |
| 8 | + n_layers: 4 | |
| 9 | + n_heads: 6 | |
| 10 | + n_kv_heads: 2 | |
| 11 | + ffn_width: 512 | |
| 12 | + max_seq_len: 320 | |
| 13 | + w: 64 | |
| 14 | + ffn_w: 192 | |
| 15 | + n_slots: 16 | |
| 16 | + n_editor_heads: 2 | |
| 17 | + r_max: 8 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint | |
| 20 | + freeze_backbone: true | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + - dev_template | |
| 30 | + - dev_depth | |
| 31 | + - dev_compose | |
| 32 | +budgets: | |
| 33 | +- 0 | |
| 34 | +- 1 | |
| 35 | +- 2 | |
| 36 | +- 4 | |
| 37 | +- 8 | |
| 38 | +budget_probs: | |
| 39 | +- 0.2 | |
| 40 | +- 0.2 | |
| 41 | +- 0.2 | |
| 42 | +- 0.2 | |
| 43 | +- 0.2 | |
| 44 | +budget_per: position | |
| 45 | +train: | |
| 46 | + batch_size: 32 | |
| 47 | + seq_len: 320 | |
| 48 | + steps: 3000 | |
| 49 | + lr: 0.001 | |
| 50 | + warmup: 100 | |
| 51 | + weight_decay: 0.1 | |
| 52 | + grad_clip: 1.0 | |
| 53 | + log_every: 50 | |
| 54 | + eval_every: 500 | |
| 55 | + eval_batches: 12 | |
| 56 | + ckpt_every: 1500 | |
| 57 | +em_budgets: | |
| 58 | +- 0 | |
| 59 | +- 2 | |
| 60 | +- 8 | |
| 61 | +em_max_items: 400 | |
| 62 | +model_overrides: | |
| 63 | + readout_zero_init: false | |
| 64 | + readout_init_std: 0.2 | |
| 65 | +baseline_kwargs: | |
| 66 | + ffn_w: 640 | |
added
configs/experiments/p4_joint_mixed_w64.yaml
+64 −0
@@ -0,0 +1,64 @@ | ||
| 1 | +name: p4_joint_mixed_w64 | |
| 2 | +seed: 1 | |
| 3 | +mode: prisme | |
| 4 | +model: | |
| 5 | + name: tiny_p4 | |
| 6 | + vocab_size: 128 | |
| 7 | + d_model: 192 | |
| 8 | + n_layers: 4 | |
| 9 | + n_heads: 6 | |
| 10 | + n_kv_heads: 2 | |
| 11 | + ffn_width: 512 | |
| 12 | + max_seq_len: 320 | |
| 13 | + w: 64 | |
| 14 | + ffn_w: 192 | |
| 15 | + n_slots: 16 | |
| 16 | + n_editor_heads: 2 | |
| 17 | + r_max: 8 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint | |
| 20 | + freeze_backbone: false | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + - dev_template | |
| 30 | + - dev_depth | |
| 31 | + - dev_compose | |
| 32 | +budgets: | |
| 33 | +- 0 | |
| 34 | +- 1 | |
| 35 | +- 2 | |
| 36 | +- 4 | |
| 37 | +- 8 | |
| 38 | +budget_probs: | |
| 39 | +- 0.2 | |
| 40 | +- 0.2 | |
| 41 | +- 0.2 | |
| 42 | +- 0.2 | |
| 43 | +- 0.2 | |
| 44 | +budget_per: position | |
| 45 | +train: | |
| 46 | + batch_size: 32 | |
| 47 | + seq_len: 320 | |
| 48 | + steps: 3000 | |
| 49 | + lr: 0.0003 | |
| 50 | + warmup: 100 | |
| 51 | + weight_decay: 0.1 | |
| 52 | + grad_clip: 1.0 | |
| 53 | + log_every: 50 | |
| 54 | + eval_every: 500 | |
| 55 | + eval_batches: 12 | |
| 56 | + ckpt_every: 1500 | |
| 57 | +em_budgets: | |
| 58 | +- 0 | |
| 59 | +- 2 | |
| 60 | +- 8 | |
| 61 | +em_max_items: 400 | |
| 62 | +model_overrides: | |
| 63 | + readout_zero_init: false | |
| 64 | + readout_init_std: 0.2 | |
added
configs/experiments/p4_overfit64_b8.yaml
+55 −0
@@ -0,0 +1,55 @@ | ||
| 1 | +name: p4_overfit64_b8 | |
| 2 | +seed: 1 | |
| 3 | +mode: prisme | |
| 4 | +model: | |
| 5 | + name: tiny_p4 | |
| 6 | + vocab_size: 128 | |
| 7 | + d_model: 192 | |
| 8 | + n_layers: 4 | |
| 9 | + n_heads: 6 | |
| 10 | + n_kv_heads: 2 | |
| 11 | + ffn_width: 512 | |
| 12 | + max_seq_len: 320 | |
| 13 | + w: 64 | |
| 14 | + ffn_w: 192 | |
| 15 | + n_slots: 16 | |
| 16 | + n_editor_heads: 2 | |
| 17 | + r_max: 8 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint | |
| 20 | + freeze_backbone: true | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + max_train_items: 64 | |
| 30 | +budgets: | |
| 31 | +- 8 | |
| 32 | +budget_probs: | |
| 33 | +- 1.0 | |
| 34 | +budget_per: position | |
| 35 | +train: | |
| 36 | + batch_size: 32 | |
| 37 | + seq_len: 320 | |
| 38 | + steps: 600 | |
| 39 | + lr: 0.001 | |
| 40 | + warmup: 100 | |
| 41 | + weight_decay: 0.1 | |
| 42 | + grad_clip: 1.0 | |
| 43 | + log_every: 50 | |
| 44 | + eval_every: 100 | |
| 45 | + eval_batches: 4 | |
| 46 | + ckpt_every: 1500 | |
| 47 | +em_budgets: | |
| 48 | +- 0 | |
| 49 | +- 2 | |
| 50 | +- 8 | |
| 51 | +em_max_items: 400 | |
| 52 | +model_overrides: | |
| 53 | + readout_zero_init: false | |
| 54 | + readout_init_std: 0.5 | |
| 55 | +final_exact_match: false | |
added
configs/experiments/p4v2_dense.yaml
+64 −0
@@ -0,0 +1,64 @@ | ||
| 1 | +seed: 1 | |
| 2 | +model: | |
| 3 | + name: tiny_p4v2 | |
| 4 | + vocab_size: 128 | |
| 5 | + d_model: 256 | |
| 6 | + n_layers: 6 | |
| 7 | + n_heads: 8 | |
| 8 | + n_kv_heads: 4 | |
| 9 | + ffn_width: 768 | |
| 10 | + max_seq_len: 320 | |
| 11 | + w: 64 | |
| 12 | + ffn_w: 192 | |
| 13 | + n_slots: 16 | |
| 14 | + n_editor_heads: 2 | |
| 15 | + r_max: 8 | |
| 16 | + readout_zero_init: false | |
| 17 | + readout_init_std: 0.2 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_v2_seed01_20261029233624/checkpoint | |
| 20 | + freeze_backbone: true | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + - dev_template | |
| 30 | + - dev_depth | |
| 31 | + - dev_compose | |
| 32 | +budgets: | |
| 33 | +- 0 | |
| 34 | +- 1 | |
| 35 | +- 2 | |
| 36 | +- 4 | |
| 37 | +- 8 | |
| 38 | +budget_probs: | |
| 39 | +- 0.2 | |
| 40 | +- 0.2 | |
| 41 | +- 0.2 | |
| 42 | +- 0.2 | |
| 43 | +- 0.2 | |
| 44 | +budget_per: position | |
| 45 | +train: | |
| 46 | + batch_size: 32 | |
| 47 | + seq_len: 320 | |
| 48 | + steps: 3000 | |
| 49 | + lr: 0.001 | |
| 50 | + warmup: 100 | |
| 51 | + weight_decay: 0.1 | |
| 52 | + grad_clip: 1.0 | |
| 53 | + log_every: 50 | |
| 54 | + eval_every: 500 | |
| 55 | + eval_batches: 12 | |
| 56 | + ckpt_every: 3000 | |
| 57 | +em_budgets: | |
| 58 | +- 0 | |
| 59 | +- 1 | |
| 60 | +- 2 | |
| 61 | +- 8 | |
| 62 | +em_max_items: 1000 | |
| 63 | +name: p4v2_dense | |
| 64 | +mode: dense | |
added
configs/experiments/p4v2_gr.yaml
+66 −0
@@ -0,0 +1,66 @@ | ||
| 1 | +seed: 1 | |
| 2 | +model: | |
| 3 | + name: tiny_p4v2 | |
| 4 | + vocab_size: 128 | |
| 5 | + d_model: 256 | |
| 6 | + n_layers: 6 | |
| 7 | + n_heads: 8 | |
| 8 | + n_kv_heads: 4 | |
| 9 | + ffn_width: 768 | |
| 10 | + max_seq_len: 320 | |
| 11 | + w: 64 | |
| 12 | + ffn_w: 192 | |
| 13 | + n_slots: 16 | |
| 14 | + n_editor_heads: 2 | |
| 15 | + r_max: 8 | |
| 16 | + readout_zero_init: false | |
| 17 | + readout_init_std: 0.2 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_v2_seed01_20261029233624/checkpoint | |
| 20 | + freeze_backbone: true | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + - dev_template | |
| 30 | + - dev_depth | |
| 31 | + - dev_compose | |
| 32 | +budgets: | |
| 33 | +- 0 | |
| 34 | +- 1 | |
| 35 | +- 2 | |
| 36 | +- 4 | |
| 37 | +- 8 | |
| 38 | +budget_probs: | |
| 39 | +- 0.2 | |
| 40 | +- 0.2 | |
| 41 | +- 0.2 | |
| 42 | +- 0.2 | |
| 43 | +- 0.2 | |
| 44 | +budget_per: position | |
| 45 | +train: | |
| 46 | + batch_size: 32 | |
| 47 | + seq_len: 320 | |
| 48 | + steps: 3000 | |
| 49 | + lr: 0.001 | |
| 50 | + warmup: 100 | |
| 51 | + weight_decay: 0.1 | |
| 52 | + grad_clip: 1.0 | |
| 53 | + log_every: 50 | |
| 54 | + eval_every: 500 | |
| 55 | + eval_batches: 12 | |
| 56 | + ckpt_every: 3000 | |
| 57 | +em_budgets: | |
| 58 | +- 0 | |
| 59 | +- 1 | |
| 60 | +- 2 | |
| 61 | +- 8 | |
| 62 | +em_max_items: 1000 | |
| 63 | +name: p4v2_gr | |
| 64 | +mode: global_recurrence | |
| 65 | +baseline_kwargs: | |
| 66 | + ffn_w: 640 | |
added
configs/experiments/p4v2_prisme.yaml
+64 −0
@@ -0,0 +1,64 @@ | ||
| 1 | +seed: 1 | |
| 2 | +model: | |
| 3 | + name: tiny_p4v2 | |
| 4 | + vocab_size: 128 | |
| 5 | + d_model: 256 | |
| 6 | + n_layers: 6 | |
| 7 | + n_heads: 8 | |
| 8 | + n_kv_heads: 4 | |
| 9 | + ffn_width: 768 | |
| 10 | + max_seq_len: 320 | |
| 11 | + w: 64 | |
| 12 | + ffn_w: 192 | |
| 13 | + n_slots: 16 | |
| 14 | + n_editor_heads: 2 | |
| 15 | + r_max: 8 | |
| 16 | + readout_zero_init: false | |
| 17 | + readout_init_std: 0.2 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_v2_seed01_20261029233624/checkpoint | |
| 20 | + freeze_backbone: true | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + - dev_template | |
| 30 | + - dev_depth | |
| 31 | + - dev_compose | |
| 32 | +budgets: | |
| 33 | +- 0 | |
| 34 | +- 1 | |
| 35 | +- 2 | |
| 36 | +- 4 | |
| 37 | +- 8 | |
| 38 | +budget_probs: | |
| 39 | +- 0.2 | |
| 40 | +- 0.2 | |
| 41 | +- 0.2 | |
| 42 | +- 0.2 | |
| 43 | +- 0.2 | |
| 44 | +budget_per: position | |
| 45 | +train: | |
| 46 | + batch_size: 32 | |
| 47 | + seq_len: 320 | |
| 48 | + steps: 3000 | |
| 49 | + lr: 0.001 | |
| 50 | + warmup: 100 | |
| 51 | + weight_decay: 0.1 | |
| 52 | + grad_clip: 1.0 | |
| 53 | + log_every: 50 | |
| 54 | + eval_every: 500 | |
| 55 | + eval_batches: 12 | |
| 56 | + ckpt_every: 3000 | |
| 57 | +em_budgets: | |
| 58 | +- 0 | |
| 59 | +- 1 | |
| 60 | +- 2 | |
| 61 | +- 8 | |
| 62 | +em_max_items: 1000 | |
| 63 | +name: p4v2_prisme | |
| 64 | +mode: prisme | |
modified
configs/experiments/routing_p4.yaml
+1 −1
@@ -1,7 +1,7 @@ | ||
| 1 | 1 | # Routing comparison (tiny scale) on dev splits with the LOCKED controller checkpoint. |
| 2 | 2 | seed: 1 |
| 3 | 3 | mode: prisme |
| 4 | −model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512, | |
| 4 | +model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512, readout_zero_init: false, readout_init_std: 0.2, readout_scale: 5.251, | |
| 5 | 5 | max_seq_len: 320, w: 64, ffn_w: 192, n_slots: 16, n_editor_heads: 2, r_max: 8} |
| 6 | 6 | checkpoint: CHECKPOINT_DIR |
| 7 | 7 | data: {kind: synthetic, tokenizer: char, splits_dir: data/splits/synthetic_v1, loss_span: answer, split: dev_iid} |
added
evaluation/label_analysis.py
+91 −0
@@ -0,0 +1,91 @@ | ||
| 1 | +#!/usr/bin/env python | |
| 2 | +"""Descriptive analysis of counterfactual labels: gain distribution, harm rate, variance decomposition of the | |
| 3 | +exhaustive 24-action audit into state and action components, per-action mean gain, oracle vs random vs STOP, | |
| 4 | +and the dependence on the prefix depth. Usage: python -m evaluation.label_analysis <labels_dir> [--out results/analyzed/labels_<name>.json] | |
| 5 | +""" | |
| 6 | +from __future__ import annotations | |
| 7 | + | |
| 8 | +import argparse, json, sys | |
| 9 | +from pathlib import Path | |
| 10 | + | |
| 11 | +import numpy as np | |
| 12 | +import torch | |
| 13 | + | |
| 14 | + | |
| 15 | +def analyse(path: Path) -> dict: | |
| 16 | + d = torch.load(path, weights_only=False) | |
| 17 | + recs = d["records"] | |
| 18 | + g_all, r_pre_all = [], [] | |
| 19 | + aud = [] | |
| 20 | + for r in recs: | |
| 21 | + v = r["valid"] | |
| 22 | + g_all.append(r["gain"][v].numpy().ravel()); r_pre_all.append(np.full(int(v.sum()) * 4, r["r_pre"])) | |
| 23 | + if "all_gains" in r: | |
| 24 | + aud.append((r["all_gains"][v].numpy(), r["base_loss"][v].numpy(), r["r_pre"])) | |
| 25 | + g = np.concatenate(g_all); rp = np.concatenate(r_pre_all) | |
| 26 | + out = {"n_labels": int(g.size), "mean": float(g.mean()), "std": float(g.std()), "median": float(np.median(g)), | |
| 27 | + "frac_pos": float((g > 0).mean()), "frac_neg": float((g < 0).mean()), "frac_abs_gt_0.01": float((np.abs(g) > 0.01).mean()), | |
| 28 | + "frac_abs_gt_0.1": float((np.abs(g) > 0.1).mean()), "q05": float(np.quantile(g, 0.05)), "q95": float(np.quantile(g, 0.95)), | |
| 29 | + "by_prefix_depth": {int(k): {"mean": float(g[rp == k].mean()), "frac_pos": float((g[rp == k] > 0).mean()), "n": int((rp == k).sum())} for k in np.unique(rp)}} | |
| 30 | + if aud: | |
| 31 | + G = np.concatenate([a for a, _, _ in aud]) # [n_states, A] | |
| 32 | + base = np.concatenate([b for _, b, _ in aud]) | |
| 33 | + A = G.shape[1] | |
| 34 | + state_mean = G.mean(1, keepdims=True) | |
| 35 | + total_var = G.var() | |
| 36 | + var_state = state_mean.var() # between-state variance of the mean gain | |
| 37 | + var_action_within = (G - state_mean).var() # within-state (action-specific) variance | |
| 38 | + best = G.max(1); best_clip = np.clip(best, 0, None) | |
| 39 | + rand = G.mean(1) | |
| 40 | + out["audit"] = { | |
| 41 | + "n_states": int(G.shape[0]), "n_actions": int(A), | |
| 42 | + "var_total": float(total_var), "var_between_states": float(var_state), "var_within_state_actions": float(var_action_within), | |
| 43 | + "within_share": float(var_action_within / max(total_var, 1e-12)), | |
| 44 | + "oracle_one_step_mean": float(best_clip.mean()), "random_action_mean": float(rand.mean()), | |
| 45 | + "stop_mean": 0.0, "frac_states_any_positive": float((best > 0).mean()), | |
| 46 | + "frac_states_all_positive": float((G > 0).all(1).mean()), "frac_states_all_negative": float((G < 0).all(1).mean()), | |
| 47 | + "mean_rank_corr_between_actions": None, | |
| 48 | + "per_action_mean": G.mean(0).tolist(), "per_action_frac_best": np.bincount(G.argmax(1), minlength=A).tolist(), | |
| 49 | + "base_loss_mean": float(base.mean()), | |
| 50 | + "oracle_gain_relative_to_base": float(best_clip.mean() / max(base.mean(), 1e-9)), | |
| 51 | + "single_vs_pair_mean": {"single": float(G[:, :16].mean()), "pair": float(G[:, 16:].mean())}, | |
| 52 | + "winners_curse_check_split_half": None, | |
| 53 | + } | |
| 54 | + # winner's-curse check: pick the best action on half the... not possible per-state (one label per action); | |
| 55 | + # instead measure how often the best action of state i is also the best of a *different* state with the | |
| 56 | + # same prefix depth (action consistency) vs chance 1/A. | |
| 57 | + am = G.argmax(1) | |
| 58 | + consist = (am[:-1] == am[1:]).mean() | |
| 59 | + out["audit"]["consecutive_state_best_action_agreement"] = float(consist) | |
| 60 | + # decomposition of the one-step oracle: continue/stop on the state mean vs choosing the best action | |
| 61 | + state_level = np.clip(G.mean(1), 0, None) | |
| 62 | + out["audit"]["oracle_state_level_mean"] = float(state_level.mean()) # STOP unless the average action helps | |
| 63 | + out["audit"]["oracle_action_selection_increment"] = float(best_clip.mean() - state_level.mean()) | |
| 64 | + out["audit"]["random_with_perfect_stop_mean"] = float(state_level.mean()) | |
| 65 | + out["audit"]["chance_agreement"] = 1.0 / A | |
| 66 | + # relation between base loss and oracle gain | |
| 67 | + hi = base > np.median(base) | |
| 68 | + out["audit"]["oracle_gain_high_vs_low_base_loss"] = {"high": float(best_clip[hi].mean()), "low": float(best_clip[~hi].mean())} | |
| 69 | + return out | |
| 70 | + | |
| 71 | + | |
| 72 | +def main(argv=None): | |
| 73 | + ap = argparse.ArgumentParser(description=__doc__) | |
| 74 | + ap.add_argument("labels_dir"); ap.add_argument("--out", default=None) | |
| 75 | + args = ap.parse_args(argv) | |
| 76 | + res = {} | |
| 77 | + for part in ["train", "calib"]: | |
| 78 | + p = Path(args.labels_dir) / f"labels_{part}.pt" | |
| 79 | + if p.exists(): | |
| 80 | + res[part] = analyse(p) | |
| 81 | + print(part, json.dumps({k: v for k, v in res[part].items() if k != "audit"}, indent=None)[:600]) | |
| 82 | + if "audit" in res[part]: | |
| 83 | + a = res[part]["audit"] | |
| 84 | + print(" audit:", {k: (round(v, 5) if isinstance(v, float) else v) for k, v in a.items() if k not in ("per_action_mean", "per_action_frac_best")}) | |
| 85 | + out = Path(args.out or f"results/analyzed/labels_{Path(args.labels_dir).name}.json") | |
| 86 | + out.parent.mkdir(parents=True, exist_ok=True); out.write_text(json.dumps(res, indent=1)) | |
| 87 | + print("wrote", out) | |
| 88 | + | |
| 89 | + | |
| 90 | +if __name__ == "__main__": | |
| 91 | + main() | |
added
evaluation/perplexity.py
+36 −0
@@ -0,0 +1,36 @@ | ||
| 1 | +"""Held-out token-level cross-entropy / perplexity at fixed forced budgets and under the learned controller.""" | |
| 2 | +from __future__ import annotations | |
| 3 | + | |
| 4 | +import math | |
| 5 | + | |
| 6 | +import numpy as np | |
| 7 | +import torch | |
| 8 | + | |
| 9 | +from prisme.losses import token_nll | |
| 10 | +from training.train_editor import sample_forced_actions | |
| 11 | + | |
| 12 | + | |
| 13 | +@torch.no_grad() | |
| 14 | +def perplexity_by_policy(model, cfg, batches_fn, device, cost, budgets=(0, 1, 2, 4, 8), controller=True, seed=0): | |
| 15 | + model.eval() | |
| 16 | + res = {} | |
| 17 | + for b in budgets: | |
| 18 | + rng = np.random.default_rng(seed) | |
| 19 | + tot, n = 0.0, 0 | |
| 20 | + for x, y in batches_fn(): | |
| 21 | + if b == 0 or not cfg.use_workspace: | |
| 22 | + logits = model(x, None)["logits"] | |
| 23 | + else: | |
| 24 | + acts, _ = sample_forced_actions(cfg, x.shape[0], x.shape[1], [b], [1.0], rng, device) | |
| 25 | + logits = model(x, acts)["logits"] | |
| 26 | + nll = token_nll(logits, y); v = y != -100 | |
| 27 | + tot += float(nll[v].sum()); n += int(v.sum()) | |
| 28 | + res[f"forced_b{b}"] = {"loss": tot / n, "ppl": math.exp(tot / n), "n_tokens": n} | |
| 29 | + if controller and cfg.use_workspace: | |
| 30 | + tot, n, ed = 0.0, 0, 0.0 | |
| 31 | + for x, y in batches_fn(): | |
| 32 | + out = model.forward_policy(x, cfg.r_max, cost) | |
| 33 | + nll = token_nll(out["logits"], y); v = y != -100 | |
| 34 | + tot += float(nll[v].sum()); n += int(v.sum()); ed += float(out["n_edits"][v].float().sum()) | |
| 35 | + res["controller"] = {"loss": tot / n, "ppl": math.exp(tot / n), "n_tokens": n, "edits_per_token": ed / n} | |
| 36 | + return res | |
added
evaluation/run.py
+92 −0
@@ -0,0 +1,92 @@ | ||
| 1 | +#!/usr/bin/env python | |
| 2 | +"""Locked evaluation driver. Reads an evaluation YAML listing systems (checkpoints + policies) and splits, refuses | |
| 3 | +to read any split whose name starts with `test` unless protocol/locked_test_manifest.json has "unsealed": true, | |
| 4 | +and writes one JSON per (system, split) under results/raw/ with full provenance. | |
| 5 | + | |
| 6 | +Usage: python -m evaluation.run --config configs/experiments/locked_evaluation.yaml [--device mps] [--only NAME] | |
| 7 | +""" | |
| 8 | +from __future__ import annotations | |
| 9 | + | |
| 10 | +import argparse, json, sys, time | |
| 11 | +from pathlib import Path | |
| 12 | + | |
| 13 | +import numpy as np | |
| 14 | +import torch | |
| 15 | + | |
| 16 | +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) | |
| 17 | +from prisme.flops import action_cost_proxy # noqa: E402 | |
| 18 | +from training.common import TokenBin, load_checkpoint, load_yaml, pick_device, seed_all, git_sha, sha256_file # noqa: E402 | |
| 19 | +from training.train_editor import build_model, sample_forced_actions # noqa: E402 | |
| 20 | +from data.synthetic_dataset import AnyTokenizer, SyntheticBatches, exact_match, load_items # noqa: E402 | |
| 21 | +from evaluation.perplexity import perplexity_by_policy # noqa: E402 | |
| 22 | + | |
| 23 | + | |
| 24 | +def check_lock(split: str): | |
| 25 | + if split.startswith("test"): | |
| 26 | + man = Path("protocol/locked_test_manifest.json") | |
| 27 | + ok = man.exists() and json.loads(man.read_text()).get("unsealed", False) | |
| 28 | + if not ok: | |
| 29 | + raise SystemExit(f"REFUSED: split '{split}' is locked (protocol/locked_test_manifest.json not unsealed)") | |
| 30 | + | |
| 31 | + | |
| 32 | +def main(argv=None) -> int: | |
| 33 | + ap = argparse.ArgumentParser(description=__doc__) | |
| 34 | + ap.add_argument("--config", required=True) | |
| 35 | + ap.add_argument("--device", default="auto") | |
| 36 | + ap.add_argument("--only", default=None) | |
| 37 | + args = ap.parse_args(argv) | |
| 38 | + exp = load_yaml(args.config); device = pick_device(args.device) | |
| 39 | + out_dir = Path(exp.get("out_dir", "results/raw")); out_dir.mkdir(parents=True, exist_ok=True) | |
| 40 | + for sysname, sysspec in exp["systems"].items(): | |
| 41 | + if args.only and sysname != args.only: | |
| 42 | + continue | |
| 43 | + seed_all(sysspec.get("seed", 1)) | |
| 44 | + cfg, model = build_model(sysspec); model = model.to(device).eval() | |
| 45 | + meta = load_checkpoint(sysspec["checkpoint"], model, strict=False) | |
| 46 | + for split in exp["splits"]: | |
| 47 | + check_lock(split) | |
| 48 | + t0 = time.time() | |
| 49 | + data = exp["data"] | |
| 50 | + rec = {"system": sysname, "split": split, "checkpoint": sysspec["checkpoint"], "checkpoint_sha256": meta.get("weights_sha256"), | |
| 51 | + "config_sha256": cfg.sha256(), "code_commit": git_sha(), "device": str(device), "precision": "fp32", "evaluated_at": time.strftime("%Y-%m-%dT%H:%M:%S%z")} | |
| 52 | + if data["kind"] == "tokens": | |
| 53 | + tb = TokenBin(Path(data["dir"]) / f"{split}.bin", exp["seq_len"]) | |
| 54 | + rec["data_sha256"] = sha256_file(Path(data["dir"]) / "manifest.json") | |
| 55 | + cost = torch.tensor(action_cost_proxy(cfg, exp["seq_len"]), device=device) | |
| 56 | + rec["perplexity"] = perplexity_by_policy(model, cfg, lambda: tb.sequential_batches(exp["batch_size"], device, exp.get("n_batches")), | |
| 57 | + device, cost, budgets=exp.get("budgets", [0, 1, 2, 4, 8]), controller=sysspec.get("controller", True)) | |
| 58 | + else: | |
| 59 | + tok = AnyTokenizer(data.get("tokenizer", "char")) | |
| 60 | + items = load_items(Path(data["splits_dir"]) / f"{split}.jsonl") | |
| 61 | + rec["data_sha256"] = sha256_file(Path(data["splits_dir"]) / "manifest.json") | |
| 62 | + sb = SyntheticBatches(items, tok, exp["seq_len"]) | |
| 63 | + cost = torch.tensor(action_cost_proxy(cfg, exp["seq_len"]), device=device) | |
| 64 | + rec["exact_match"] = {} | |
| 65 | + policies = sysspec.get("policies", ["budget0"]) | |
| 66 | + for pol in policies: | |
| 67 | + rng = np.random.default_rng(sysspec.get("seed", 1) + 7) | |
| 68 | + if pol == "budget0": | |
| 69 | + em = exact_match(model, sb, tok, device, 0, batch_size=exp["batch_size"], max_items=exp.get("max_items")) | |
| 70 | + elif pol.startswith("forced"): | |
| 71 | + b = int(pol.split("_b")[1]) | |
| 72 | + em = exact_match(model, sb, tok, device, b, batch_size=exp["batch_size"], max_items=exp.get("max_items"), | |
| 73 | + forced_actions_fn=lambda B_, n, b=b: sample_forced_actions(cfg, B_, n, [b], [1.0], rng, device)[0]) | |
| 74 | + elif pol == "controller": | |
| 75 | + em = exact_match(model, sb, tok, device, cfg.r_max, cost=cost, batch_size=exp["batch_size"], max_items=exp.get("max_items")) | |
| 76 | + elif pol.startswith("controller_lam"): | |
| 77 | + lam = float(pol.split("lam")[1]) | |
| 78 | + em = exact_match(model, sb, tok, device, cfg.r_max, cost=cost, batch_size=exp["batch_size"], max_items=exp.get("max_items"), lambda_cost=lam) | |
| 79 | + else: | |
| 80 | + raise ValueError(pol) | |
| 81 | + recs = em.pop("records") | |
| 82 | + rec["exact_match"][pol] = em | |
| 83 | + (out_dir / f"items_{sysname}_{split}_{pol}.json").write_text(json.dumps(recs)) | |
| 84 | + print(f"{sysname} {split} {pol}: acc {em['accuracy']:.4f} n={em['n']} edits/tok {em['edits_per_token']:.2f}", flush=True) | |
| 85 | + rec["seconds"] = round(time.time() - t0, 1) | |
| 86 | + (out_dir / f"eval_{sysname}_{split}.json").write_text(json.dumps(rec, indent=1)) | |
| 87 | + print("wrote", out_dir / f"eval_{sysname}_{split}.json", flush=True) | |
| 88 | + return 0 | |
| 89 | + | |
| 90 | + | |
| 91 | +if __name__ == "__main__": | |
| 92 | + raise SystemExit(main()) | |
added
experiments/logs/latency_p4_mps.log
+27 −0
@@ -0,0 +1,27 @@ | ||
| 1 | +{"policy": "budget0", "batch": 1, "budget": 0, "median_ms": 2.8017294825986028, "p95_ms": 3.1844482582528144, "mean_ms": 2.759531239765541, "n": 96, "cache_bytes": 245760} | |
| 2 | +{"policy": "forced_single", "batch": 1, "budget": 1, "forced_k": 1, "median_ms": 5.252917006146163, "p95_ms": 6.079749495256692, "mean_ms": 5.123651044414146, "n": 96, "cache_bytes": 245760} | |
| 3 | +{"policy": "forced_pair", "batch": 1, "budget": 1, "forced_k": 2, "median_ms": 5.21864602342248, "p95_ms": 6.357603764627129, "mean_ms": 4.836453521420481, "n": 96, "cache_bytes": 245760} | |
| 4 | +{"policy": "forced_single", "batch": 1, "budget": 2, "forced_k": 1, "median_ms": 7.243604050017893, "p95_ms": 9.018541488330811, "mean_ms": 7.380335543227072, "n": 96, "cache_bytes": 245760} | |
| 5 | +{"policy": "forced_pair", "batch": 1, "budget": 2, "forced_k": 2, "median_ms": 7.464978960342705, "p95_ms": 12.41436498821713, "mean_ms": 8.055094188118042, "n": 96, "cache_bytes": 245760} | |
| 6 | +{"policy": "forced_single", "batch": 1, "budget": 4, "forced_k": 1, "median_ms": 11.287728964816779, "p95_ms": 12.40296903415583, "mean_ms": 10.915309430856723, "n": 96, "cache_bytes": 245760} | |
| 7 | +{"policy": "forced_pair", "batch": 1, "budget": 4, "forced_k": 2, "median_ms": 11.351562512572855, "p95_ms": 12.37730251159519, "mean_ms": 11.322401035916604, "n": 96, "cache_bytes": 245760} | |
| 8 | +{"policy": "forced_single", "batch": 1, "budget": 8, "forced_k": 1, "median_ms": 19.331770483404398, "p95_ms": 21.22909395257011, "mean_ms": 19.320005204159923, "n": 96, "cache_bytes": 245760} | |
| 9 | +{"policy": "forced_pair", "batch": 1, "budget": 8, "forced_k": 2, "median_ms": 18.950854428112507, "p95_ms": 20.537573233013973, "mean_ms": 19.09837671701098, "n": 96, "cache_bytes": 245760} | |
| 10 | +{"policy": "controller", "batch": 1, "budget": 8, "median_ms": 5.7201660820283, "p95_ms": 23.911051946925, "mean_ms": 9.350690118784163, "n": 96, "cache_bytes": 245760} | |
| 11 | +{"policy": "dense_forced", "batch": 1, "budget": 1, "forced_k": 1, "median_ms": 4.747916478663683, "p95_ms": 8.084010769380257, "mean_ms": 4.76325996714877, "n": 96, "cache_bytes": 245760} | |
| 12 | +{"policy": "dense_forced", "batch": 1, "budget": 4, "forced_k": 1, "median_ms": 10.018958011642098, "p95_ms": 11.016781529178843, "mean_ms": 10.038378921308322, "n": 96, "cache_bytes": 245760} | |
| 13 | +{"policy": "dense_forced", "batch": 1, "budget": 8, "forced_k": 1, "median_ms": 17.32618745882064, "p95_ms": 27.340614731656387, "mean_ms": 17.98606334341457, "n": 96, "cache_bytes": 245760} | |
| 14 | +{"policy": "budget0", "batch": 8, "budget": 0, "median_ms": 3.36672953562811, "p95_ms": 3.923937038052827, "mean_ms": 3.3720702607145845, "n": 96, "cache_bytes": 1966080} | |
| 15 | +{"policy": "forced_single", "batch": 8, "budget": 1, "forced_k": 1, "median_ms": 6.341208005324006, "p95_ms": 10.401958308648318, "mean_ms": 6.62630293663824, "n": 96, "cache_bytes": 1966080} | |
| 16 | +{"policy": "forced_pair", "batch": 8, "budget": 1, "forced_k": 2, "median_ms": 6.1652500298805535, "p95_ms": 7.15568705345504, "mean_ms": 6.102194831328234, "n": 96, "cache_bytes": 1966080} | |
| 17 | +{"policy": "forced_single", "batch": 8, "budget": 2, "forced_k": 1, "median_ms": 8.590396028012037, "p95_ms": 14.236905990401283, "mean_ms": 8.708206226098506, "n": 96, "cache_bytes": 1966080} | |
| 18 | +{"policy": "forced_pair", "batch": 8, "budget": 2, "forced_k": 2, "median_ms": 8.381562540307641, "p95_ms": 9.846666041994467, "mean_ms": 8.429156674537808, "n": 96, "cache_bytes": 1966080} | |
| 19 | +{"policy": "forced_single", "batch": 8, "budget": 4, "forced_k": 1, "median_ms": 12.963792018126696, "p95_ms": 14.19885375071317, "mean_ms": 12.95540061983047, "n": 96, "cache_bytes": 1966080} | |
| 20 | +{"policy": "forced_pair", "batch": 8, "budget": 4, "forced_k": 2, "median_ms": 13.345645973458886, "p95_ms": 20.811073016375303, "mean_ms": 13.279389810728995, "n": 96, "cache_bytes": 1966080} | |
| 21 | +{"policy": "forced_single", "batch": 8, "budget": 8, "forced_k": 1, "median_ms": 22.172499971929938, "p95_ms": 26.029041764559224, "mean_ms": 19.927763494100265, "n": 96, "cache_bytes": 1966080} | |
| 22 | +{"policy": "forced_pair", "batch": 8, "budget": 8, "forced_k": 2, "median_ms": 22.192937438376248, "p95_ms": 24.253927753306925, "mean_ms": 21.646448845179595, "n": 96, "cache_bytes": 1966080} | |
| 23 | +{"policy": "controller", "batch": 8, "budget": 8, "median_ms": 24.369208491407335, "p95_ms": 26.040082942927256, "mean_ms": 21.67698220121868, "n": 96, "cache_bytes": 1966080} | |
| 24 | +{"policy": "dense_forced", "batch": 8, "budget": 1, "forced_k": 1, "median_ms": 5.198728991672397, "p95_ms": 6.063812266802415, "mean_ms": 5.117546953745962, "n": 96, "cache_bytes": 1966080} | |
| 25 | +{"policy": "dense_forced", "batch": 8, "budget": 4, "forced_k": 1, "median_ms": 10.355145495850593, "p95_ms": 11.753489437978715, "mean_ms": 10.392837262770627, "n": 96, "cache_bytes": 1966080} | |
| 26 | +{"policy": "dense_forced", "batch": 8, "budget": 8, "forced_k": 1, "median_ms": 17.042521038092673, "p95_ms": 19.3497292639222, "mean_ms": 16.37415804725606, "n": 96, "cache_bytes": 1966080} | |
| 27 | +wrote results/raw/latency_p4_editor_v1_Ustd02_seed01_20261029235050_mps.json | |
modified
experiments/logs/p4_editor_v1_scaled.log
+95 −0
@@ -3,3 +3,98 @@ readout_scale set to backbone residual rms = 5.251 | ||
| 3 | 3 | /Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/training/train_editor.py:55: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /Users/runner/work/pytorch/pytorch/torch/csrc/utils/tensor_numpy.cpp:214.) |
| 4 | 4 | return torch.from_numpy(acts).to(device), torch.from_numpy(np.ascontiguousarray(b)).to(device) |
| 5 | 5 | step 0 loss 0.2574 lr 1.00e-05 gn 0.02 (1s) |
| 6 | +step 50 loss 0.2963 lr 5.10e-04 gn 0.02 (35s) | |
| 7 | +step 100 loss 0.2928 lr 1.00e-03 gn 0.03 (68s) | |
| 8 | +step 150 loss 0.2752 lr 9.99e-04 gn 0.04 (101s) | |
| 9 | +step 200 loss 0.2912 lr 9.97e-04 gn 0.03 (134s) | |
| 10 | +step 250 loss 0.2685 lr 9.94e-04 gn 0.02 (168s) | |
| 11 | +step 300 loss 0.2521 lr 9.89e-04 gn 0.02 (200s) | |
| 12 | +step 350 loss 0.3760 lr 9.84e-04 gn 0.10 (234s) | |
| 13 | +step 400 loss 0.2934 lr 9.76e-04 gn 0.03 (266s) | |
| 14 | +step 450 loss 0.2019 lr 9.68e-04 gn 0.06 (299s) | |
| 15 | + val loss by budget: {0: 0.4144, 1: 0.4139, 2: 0.4133, 4: 0.4131, 8: 0.4149} | |
| 16 | +step 500 loss 0.3621 lr 9.58e-04 gn 0.03 (327s) | |
| 17 | +step 550 loss 0.2949 lr 9.48e-04 gn 0.04 (349s) | |
| 18 | +step 600 loss 0.3686 lr 9.36e-04 gn 0.07 (372s) | |
| 19 | +step 650 loss 0.3048 lr 9.22e-04 gn 0.03 (407s) | |
| 20 | +step 700 loss 0.3155 lr 9.08e-04 gn 0.04 (440s) | |
| 21 | +step 750 loss 0.2576 lr 8.93e-04 gn 0.02 (478s) | |
| 22 | +step 800 loss 0.3423 lr 8.77e-04 gn 0.04 (527s) | |
| 23 | +step 850 loss 0.3550 lr 8.59e-04 gn 0.07 (579s) | |
| 24 | +step 900 loss 0.3444 lr 8.41e-04 gn 0.06 (627s) | |
| 25 | +step 950 loss 0.3196 lr 8.22e-04 gn 0.05 (671s) | |
| 26 | + val loss by budget: {0: 0.4144, 1: 0.4129, 2: 0.4124, 4: 0.4123, 8: 0.4146} | |
| 27 | +step 1000 loss 0.3007 lr 8.03e-04 gn 0.06 (751s) | |
| 28 | +step 1050 loss 0.2752 lr 7.82e-04 gn 0.10 (820s) | |
| 29 | +step 1100 loss 0.2404 lr 7.61e-04 gn 0.08 (889s) | |
| 30 | +step 1150 loss 0.3453 lr 7.39e-04 gn 0.05 (948s) | |
| 31 | +step 1200 loss 0.3205 lr 7.17e-04 gn 0.08 (987s) | |
| 32 | +step 1250 loss 0.3766 lr 6.94e-04 gn 0.11 (1018s) | |
| 33 | +step 1300 loss 0.2507 lr 6.70e-04 gn 0.03 (1053s) | |
| 34 | +step 1350 loss 0.2750 lr 6.47e-04 gn 0.09 (1087s) | |
| 35 | +step 1400 loss 0.2685 lr 6.23e-04 gn 0.09 (1124s) | |
| 36 | +step 1450 loss 0.3713 lr 5.99e-04 gn 0.09 (1158s) | |
| 37 | + val loss by budget: {0: 0.4144, 1: 0.412, 2: 0.412, 4: 0.4141, 8: 0.4231} | |
| 38 | +step 1500 loss 0.2672 lr 5.74e-04 gn 0.11 (1197s) | |
| 39 | +step 1550 loss 0.3123 lr 5.50e-04 gn 0.10 (1231s) | |
| 40 | +step 1600 loss 0.2820 lr 5.26e-04 gn 0.09 (1259s) | |
| 41 | +step 1650 loss 0.3476 lr 5.01e-04 gn 0.06 (1289s) | |
| 42 | +step 1700 loss 0.3768 lr 4.77e-04 gn 0.09 (1318s) | |
| 43 | +step 1750 loss 0.2676 lr 4.53e-04 gn 0.05 (1347s) | |
| 44 | +step 1800 loss 0.2471 lr 4.30e-04 gn 0.05 (1382s) | |
| 45 | +step 1850 loss 0.2802 lr 4.06e-04 gn 0.10 (1418s) | |
| 46 | +step 1900 loss 0.2870 lr 3.83e-04 gn 0.07 (1452s) | |
| 47 | +step 1950 loss 0.2158 lr 3.61e-04 gn 0.09 (1488s) | |
| 48 | + val loss by budget: {0: 0.4144, 1: 0.4119, 2: 0.4129, 4: 0.4139, 8: 0.4241} | |
| 49 | +step 2000 loss 0.2418 lr 3.39e-04 gn 0.09 (1524s) | |
| 50 | +step 2050 loss 0.2704 lr 3.18e-04 gn 0.12 (1553s) | |
| 51 | +step 2100 loss 0.2859 lr 2.97e-04 gn 0.15 (1583s) | |
| 52 | +step 2150 loss 0.2684 lr 2.78e-04 gn 0.06 (1611s) | |
| 53 | +step 2200 loss 0.2357 lr 2.59e-04 gn 0.11 (1640s) | |
| 54 | +step 2250 loss 0.2331 lr 2.41e-04 gn 0.06 (1674s) | |
| 55 | +step 2300 loss 0.2750 lr 2.23e-04 gn 0.08 (1709s) | |
| 56 | +step 2350 loss 0.2091 lr 2.07e-04 gn 0.10 (1742s) | |
| 57 | +step 2400 loss 0.1803 lr 1.92e-04 gn 0.08 (1774s) | |
| 58 | +step 2450 loss 0.3071 lr 1.78e-04 gn 0.09 (1804s) | |
| 59 | + val loss by budget: {0: 0.4144, 1: 0.41, 2: 0.41, 4: 0.4111, 8: 0.4188} | |
| 60 | +step 2500 loss 0.3278 lr 1.64e-04 gn 0.13 (1842s) | |
| 61 | +step 2550 loss 0.2849 lr 1.52e-04 gn 0.09 (1877s) | |
| 62 | +step 2600 loss 0.2684 lr 1.42e-04 gn 0.12 (1911s) | |
| 63 | +step 2650 loss 0.2542 lr 1.32e-04 gn 0.16 (1946s) | |
| 64 | +step 2700 loss 0.1905 lr 1.24e-04 gn 0.08 (1978s) | |
| 65 | +step 2750 loss 0.2685 lr 1.16e-04 gn 0.13 (2007s) | |
| 66 | +step 2800 loss 0.2449 lr 1.11e-04 gn 0.10 (2034s) | |
| 67 | +step 2850 loss 0.2741 lr 1.06e-04 gn 0.17 (2059s) | |
| 68 | +step 2900 loss 0.2866 lr 1.03e-04 gn 0.13 (2084s) | |
| 69 | +step 2950 loss 0.2046 lr 1.01e-04 gn 0.07 (2104s) | |
| 70 | +step 2999 loss 0.3118 lr 1.00e-04 gn 0.08 (2120s) | |
| 71 | + val loss by budget: {0: 0.4144, 1: 0.4093, 2: 0.4092, 4: 0.411, 8: 0.4192} | |
| 72 | + exact match dev_iid budget 0: 0.4375 (n=400) by family {'fsm': 0.640625, 'bind': 0.38372093023255816, 'arith': 0.22784810126582278, 'excl': 0.27906976744186046, 'sched': 0.6941176470588235} | |
| 73 | +Traceback (most recent call last): | |
| 74 | + File "<frozen runpy>", line 198, in _run_module_as_main | |
| 75 | + File "<frozen runpy>", line 88, in _run_code | |
| 76 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/training/train_editor.py", line 186, in <module> | |
| 77 | + metrics_file=str(run.dir / "metrics.jsonl")) | |
| 78 | + ^^^^^^ | |
| 79 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/training/train_editor.py", line 173, in main | |
| 80 | + for name, vs in val_sets.items(): | |
| 81 | + ^^^^^^^^^^^^^^^^^^^^ | |
| 82 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 158, in no_grad | |
| 83 | + return func(*args, **kwargs) | |
| 84 | + ^^^^^^^^^^^^^^^^^^^^^ | |
| 85 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/data/synthetic_dataset.py", line 105, in exact_match | |
| 86 | + out = generate(model, prompt, max_new, max_budget=max_budget, cost=cost, temperature=0.0, eos_id=tok.eot_id, | |
| 87 | + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 88 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 158, in no_grad | |
| 89 | + return func(*args, **kwargs) | |
| 90 | + ^^^^^^^^^^^^^^^^^^^^^ | |
| 91 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/prisme/generation.py", line 115, in generate | |
| 92 | + logits, _, _ = step_logits(model, prompt, cache, max_budget, cost, fa, lambda_cost, kappa_risk) | |
| 93 | + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 94 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 158, in no_grad | |
| 95 | + return func(*args, **kwargs) | |
| 96 | + ^^^^^^^^^^^^^^^^^^^^^ | |
| 97 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/prisme/generation.py", line 43, in step_logits | |
| 98 | + if cfg.ctrl_use_entropy: | |
| 99 | + ^^^^^^^^^^^^^^^^^^^^ | |
| 100 | +AttributeError: 'PrismeConfig' object has no attribute 'ctrl_use_entropy' | |
added
experiments/logs/p4_overfit64_b8.log
+24 −0
@@ -0,0 +1,24 @@ | ||
| 1 | +loaded backbone from experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint missing: 35 | |
| 2 | +readout_scale set to backbone residual rms = 5.310 | |
| 3 | +/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/training/train_editor.py:55: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /Users/runner/work/pytorch/pytorch/torch/csrc/utils/tensor_numpy.cpp:214.) | |
| 4 | + return torch.from_numpy(acts).to(device), torch.from_numpy(np.ascontiguousarray(b)).to(device) | |
| 5 | +step 0 loss 0.3472 lr 1.00e-05 gn 1.32 (4s) | |
| 6 | +step 50 loss 0.1013 lr 5.10e-04 gn 0.40 (41s) | |
| 7 | + val loss by budget: {8: 0.7307} | |
| 8 | +step 100 loss 0.0165 lr 1.00e-03 gn 0.16 (90s) | |
| 9 | +step 150 loss 0.0053 lr 9.78e-04 gn 0.02 (137s) | |
| 10 | + val loss by budget: {8: 1.1005} | |
| 11 | +step 200 loss 0.0024 lr 9.14e-04 gn 0.01 (173s) | |
| 12 | +step 250 loss 0.0010 lr 8.15e-04 gn 0.00 (206s) | |
| 13 | + val loss by budget: {8: 1.3595} | |
| 14 | +step 300 loss 0.0008 lr 6.89e-04 gn 0.00 (254s) | |
| 15 | +step 350 loss 0.0004 lr 5.50e-04 gn 0.00 (300s) | |
| 16 | + val loss by budget: {8: 1.4991} | |
| 17 | +step 400 loss 0.0004 lr 4.11e-04 gn 0.00 (347s) | |
| 18 | +step 450 loss 0.0004 lr 2.85e-04 gn 0.00 (391s) | |
| 19 | + val loss by budget: {8: 1.6038} | |
| 20 | +step 500 loss 0.0006 lr 1.86e-04 gn 0.00 (437s) | |
| 21 | +step 550 loss 0.0003 lr 1.22e-04 gn 0.00 (480s) | |
| 22 | +step 599 loss 0.0003 lr 1.00e-04 gn 0.00 (517s) | |
| 23 | + val loss by budget: {8: 1.645} | |
| 24 | +run p4_overfit64_b8_seed01_20261029235111 done | |
modified
experiments/logs/p4_tiny_backbone_v2.log
+80 −0
@@ -25,3 +25,83 @@ step 1900 loss 0.2829 lr 9.74e-04 gn 0.10 (380s) | ||
| 25 | 25 | step 2000 loss 0.2506 lr 9.71e-04 gn 0.12 (400s) |
| 26 | 26 | step 2100 loss 0.2389 lr 9.67e-04 gn 0.09 (426s) |
| 27 | 27 | step 2200 loss 0.2490 lr 9.63e-04 gn 0.15 (455s) |
| 28 | +step 2300 loss 0.2360 lr 9.60e-04 gn 0.16 (484s) | |
| 29 | +step 2400 loss 0.2411 lr 9.55e-04 gn 0.09 (512s) | |
| 30 | +step 2500 loss 0.2048 lr 9.51e-04 gn 0.07 (541s) | |
| 31 | +step 2600 loss 0.2770 lr 9.47e-04 gn 0.11 (570s) | |
| 32 | +step 2700 loss 0.2219 lr 9.42e-04 gn 0.07 (600s) | |
| 33 | +step 2800 loss 0.2500 lr 9.37e-04 gn 0.11 (629s) | |
| 34 | +step 2900 loss 0.2512 lr 9.32e-04 gn 0.08 (658s) | |
| 35 | + val loss by budget: {0: 0.2491} | |
| 36 | +step 3000 loss 0.2259 lr 9.27e-04 gn 0.08 (688s) | |
| 37 | +step 3100 loss 0.2322 lr 9.22e-04 gn 0.09 (716s) | |
| 38 | +step 3200 loss 0.2616 lr 9.16e-04 gn 0.10 (743s) | |
| 39 | +step 3300 loss 0.2512 lr 9.11e-04 gn 0.09 (763s) | |
| 40 | +step 3400 loss 0.2397 lr 9.05e-04 gn 0.12 (782s) | |
| 41 | +step 3500 loss 0.2581 lr 8.99e-04 gn 0.10 (804s) | |
| 42 | +step 3600 loss 0.2327 lr 8.93e-04 gn 0.09 (833s) | |
| 43 | +step 3700 loss 0.2532 lr 8.86e-04 gn 0.09 (862s) | |
| 44 | +step 3800 loss 0.2413 lr 8.80e-04 gn 0.08 (893s) | |
| 45 | +step 3900 loss 0.2293 lr 8.73e-04 gn 0.09 (927s) | |
| 46 | + val loss by budget: {0: 0.2467} | |
| 47 | +step 4000 loss 0.2489 lr 8.66e-04 gn 0.08 (966s) | |
| 48 | +step 4100 loss 0.2392 lr 8.60e-04 gn 0.09 (1009s) | |
| 49 | +step 4200 loss 0.2620 lr 8.53e-04 gn 0.10 (1046s) | |
| 50 | +step 4300 loss 0.2340 lr 8.45e-04 gn 0.10 (1077s) | |
| 51 | +step 4400 loss 0.2159 lr 8.38e-04 gn 0.09 (1104s) | |
| 52 | +step 4500 loss 0.2391 lr 8.31e-04 gn 0.11 (1129s) | |
| 53 | +step 4600 loss 0.2459 lr 8.23e-04 gn 0.10 (1153s) | |
| 54 | +step 4700 loss 0.2321 lr 8.15e-04 gn 0.08 (1177s) | |
| 55 | +step 4800 loss 0.2316 lr 8.07e-04 gn 0.08 (1201s) | |
| 56 | +step 4900 loss 0.2222 lr 8.00e-04 gn 0.09 (1225s) | |
| 57 | + val loss by budget: {0: 0.2458} | |
| 58 | +step 5000 loss 0.2277 lr 7.91e-04 gn 0.08 (1252s) | |
| 59 | +step 5100 loss 0.2377 lr 7.83e-04 gn 0.11 (1277s) | |
| 60 | +step 5200 loss 0.2480 lr 7.75e-04 gn 0.10 (1302s) | |
| 61 | +step 5300 loss 0.2394 lr 7.67e-04 gn 0.09 (1327s) | |
| 62 | +step 5400 loss 0.2269 lr 7.58e-04 gn 0.09 (1351s) | |
| 63 | +step 5500 loss 0.2226 lr 7.50e-04 gn 0.07 (1379s) | |
| 64 | +step 5600 loss 0.2372 lr 7.41e-04 gn 0.10 (1411s) | |
| 65 | +step 5700 loss 0.2575 lr 7.32e-04 gn 0.15 (1438s) | |
| 66 | +step 5800 loss 0.2327 lr 7.23e-04 gn 0.09 (1467s) | |
| 67 | +step 5900 loss 0.2266 lr 7.14e-04 gn 0.08 (1495s) | |
| 68 | + val loss by budget: {0: 0.2471} | |
| 69 | +step 6000 loss 0.2355 lr 7.05e-04 gn 0.09 (1525s) | |
| 70 | +step 6100 loss 0.2458 lr 6.96e-04 gn 0.09 (1556s) | |
| 71 | +step 6200 loss 0.2058 lr 6.87e-04 gn 0.08 (1585s) | |
| 72 | +step 6300 loss 0.2245 lr 6.78e-04 gn 0.10 (1614s) | |
| 73 | +step 6400 loss 0.2180 lr 6.69e-04 gn 0.10 (1642s) | |
| 74 | +step 6500 loss 0.2223 lr 6.59e-04 gn 0.08 (1670s) | |
| 75 | +step 6600 loss 0.2298 lr 6.50e-04 gn 0.09 (1693s) | |
| 76 | +step 6700 loss 0.2283 lr 6.41e-04 gn 0.09 (1719s) | |
| 77 | +step 6800 loss 0.2490 lr 6.31e-04 gn 0.11 (1745s) | |
| 78 | +step 6900 loss 0.2456 lr 6.22e-04 gn 0.11 (1766s) | |
| 79 | + val loss by budget: {0: 0.2523} | |
| 80 | +step 7000 loss 0.2237 lr 6.12e-04 gn 0.10 (1798s) | |
| 81 | +step 7100 loss 0.2181 lr 6.03e-04 gn 0.10 (1827s) | |
| 82 | +step 7200 loss 0.1884 lr 5.93e-04 gn 0.10 (1857s) | |
| 83 | +step 7300 loss 0.2220 lr 5.84e-04 gn 0.09 (1884s) | |
| 84 | +step 7400 loss 0.2312 lr 5.74e-04 gn 0.10 (1914s) | |
| 85 | +step 7500 loss 0.2148 lr 5.64e-04 gn 0.10 (1942s) | |
| 86 | +step 7600 loss 0.2208 lr 5.55e-04 gn 0.09 (1966s) | |
| 87 | +step 7700 loss 0.2331 lr 5.45e-04 gn 0.11 (1992s) | |
| 88 | +step 7800 loss 0.2063 lr 5.36e-04 gn 0.11 (2017s) | |
| 89 | +step 7900 loss 0.2102 lr 5.26e-04 gn 0.10 (2040s) | |
| 90 | + val loss by budget: {0: 0.264} | |
| 91 | +step 8000 loss 0.2142 lr 5.16e-04 gn 0.11 (2066s) | |
| 92 | +step 8100 loss 0.1659 lr 5.07e-04 gn 0.08 (2095s) | |
| 93 | +step 8200 loss 0.2003 lr 4.97e-04 gn 0.13 (2124s) | |
| 94 | +step 8300 loss 0.2093 lr 4.88e-04 gn 0.12 (2152s) | |
| 95 | +step 8400 loss 0.1932 lr 4.78e-04 gn 0.11 (2181s) | |
| 96 | +step 8500 loss 0.2004 lr 4.69e-04 gn 0.12 (2207s) | |
| 97 | +step 8600 loss 0.2052 lr 4.59e-04 gn 0.13 (2233s) | |
| 98 | +step 8700 loss 0.2038 lr 4.50e-04 gn 0.12 (2261s) | |
| 99 | +step 8800 loss 0.1986 lr 4.41e-04 gn 0.13 (2290s) | |
| 100 | +step 8900 loss 0.2058 lr 4.31e-04 gn 0.12 (2321s) | |
| 101 | + val loss by budget: {0: 0.2822} | |
| 102 | +step 9000 loss 0.1914 lr 4.22e-04 gn 0.14 (2351s) | |
| 103 | +step 9100 loss 0.2039 lr 4.13e-04 gn 0.15 (2380s) | |
| 104 | +step 9200 loss 0.1904 lr 4.04e-04 gn 0.15 (2407s) | |
| 105 | +step 9300 loss 0.1848 lr 3.95e-04 gn 0.13 (2432s) | |
| 106 | +step 9400 loss 0.1662 lr 3.86e-04 gn 0.14 (2455s) | |
| 107 | +step 9500 loss 0.1821 lr 3.77e-04 gn 0.16 (2475s) | |
added
experiments/logs/routing_p4_dev_iid.log
+16 −0
@@ -0,0 +1,16 @@ | ||
| 1 | +Traceback (most recent call last): | |
| 2 | + File "<frozen runpy>", line 198, in _run_module_as_main | |
| 3 | + File "<frozen runpy>", line 88, in _run_code | |
| 4 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/evaluation/routing.py", line 123, in <module> | |
| 5 | + raise SystemExit(main()) | |
| 6 | + ^^^^^^ | |
| 7 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/evaluation/routing.py", line 73, in main | |
| 8 | + meta = load_checkpoint(exp["checkpoint"], model, strict=False) | |
| 9 | + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 10 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/training/common.py", line 174, in load_checkpoint | |
| 11 | + sd = load_file(str(path / "model.safetensors")) | |
| 12 | + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 13 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/.venv/lib/python3.12/site-packages/safetensors/torch.py", line 359, in load_file | |
| 14 | + with safe_open(filename, framework="pt", device=device, backend=backend) as f: | |
| 15 | + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 16 | +FileNotFoundError: No such file or directory: CHECKPOINT_DIR/model.safetensors | |
added
experiments/logs/setup3.log
+5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/logs/setup4.log
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/logs/setup_M4BP36.log
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/logs/setup_M4BP48.log
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/logs/setup_M5M36.log
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/logs/setup_m4mm.log
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/logs/setup_m4mn.log
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/controller_p4_hent_seed01_20261030001419/config.yaml
+83 −0
Large diff (83 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/controller_p4_hent_seed01_20261030001419/manifest.json
+18 −0
Large diff (19 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/controller_p4_hent_seed01_20261030001419/metrics.jsonl
+34 −0
Large diff (34 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/labels_p4_seed01_20261030000543/config.yaml
+72 −0
Large diff (72 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/labels_p4_seed01_20261030000543/manifest.json
+40 −0
Large diff (41 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/labels_p4_seed01_20261030000543/metrics.jsonl
+0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/checkpoint/checkpoint.json
+37 −0
Large diff (38 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/config.yaml
+107 −0
Large diff (107 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_compose_b0.json
+1 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_compose_b2.json
+1 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_compose_b8.json
+1 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_depth_b0.json
+1 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_depth_b2.json
+1 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_depth_b8.json
+1 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_iid_b0.json
+1 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_iid_b2.json
+1 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_iid_b8.json
+1 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_template_b0.json
+1 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_template_b2.json
+1 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_template_b8.json
+1 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/manifest.json
+169 −0
Large diff (170 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/metrics.jsonl
+47 −0
Large diff (47 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/p4_editor_v1_scaled_seed01_20261029234334/checkpoint/checkpoint.json
+36 −0
Large diff (37 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/p4_editor_v1_scaled_seed01_20261029234334/em_dev_iid_b0.json
+1 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
modified
experiments/runs/p4_editor_v1_scaled_seed01_20261029234334/metrics.jsonl
+68 −0
Large diff (69 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/p4_overfit64_b8_seed01_20261029235111/checkpoint/checkpoint.json
+37 −0
Large diff (38 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/p4_overfit64_b8_seed01_20261029235111/config.yaml
+98 −0
Large diff (98 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/p4_overfit64_b8_seed01_20261029235111/manifest.json
+28 −0
Large diff (29 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/p4_overfit64_b8_seed01_20261029235111/metrics.jsonl
+20 −0
Large diff (20 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/runs/p4_tiny_backbone_v2_seed01_20261029233624/checkpoint/checkpoint.json
+35 −0
Large diff (36 lines) is not loaded by default to keep the page fast.Load diff
modified
experiments/runs/p4_tiny_backbone_v2_seed01_20261029233624/manifest.json
+1 −1
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
modified
experiments/runs/p4_tiny_backbone_v2_seed01_20261029233624/metrics.jsonl
+81 −0
Large diff (84 lines) is not loaded by default to keep the page fast.Load diff
modified
literature/papers.bib
+9 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
added
paper/.latexmkrc
+5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added
paper/Makefile
+11 −0
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
added
paper/appendix/appendix.tex
+18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added
paper/appendix/failures.tex
+21 −0
Large diff (21 lines) is not loaded by default to keep the page fast.Load diff
added
paper/appendix/literature.tex
+22 −0
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff
added
paper/appendix/notation.tex
+16 −0
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
added
paper/appendix/params.tex
+16 −0
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
added
paper/appendix/prereg.tex
+59 −0
Large diff (59 lines) is not loaded by default to keep the page fast.Load diff
added
paper/appendix/proofs.tex
+46 −0
Large diff (46 lines) is not loaded by default to keep the page fast.Load diff
added
paper/appendix/results.tex
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
paper/appendix/runs.tex
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
paper/figures/architecture.tex
+36 −0
Large diff (36 lines) is not loaded by default to keep the page fast.Load diff
added
paper/main.log
+1558 −0
Large diff (1,558 lines) is not loaded by default to keep the page fast.Load diff
added
paper/main.pdf
+0 −0
Binary file not shown.
added
paper/main.tex
+104 −0
Large diff (104 lines) is not loaded by default to keep the page fast.Load diff
added
paper/references.bib
+739 −0
Large diff (739 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/abstract.tex
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/architecture.tex
+97 −0
Large diff (97 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/conclusion.tex
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/design.tex
+42 −0
Large diff (42 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/implementation.tex
+46 −0
Large diff (46 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/introduction.tex
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/learning.tex
+60 −0
Large diff (60 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/limitations.tex
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/mechanistic.tex
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/policy.tex
+32 −0
Large diff (32 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/problem.tex
+30 −0
Large diff (30 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/related.tex
+53 −0
Large diff (53 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/reproducibility.tex
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/results.tex
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/theory.tex
+48 −0
Large diff (48 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/titlepage.tex
+77 −0
Large diff (77 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/titlepage_anon.tex
+53 −0
Large diff (53 lines) is not loaded by default to keep the page fast.Load diff
added
paper/tables/configs.tex
+18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added
paper/tables/controller.tex
+8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added
paper/tables/controls.tex
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
paper/tables/families.tex
+17 −0
Large diff (17 lines) is not loaded by default to keep the page fast.Load diff
added
paper/tables/numbers.tex
+24 −0
Large diff (24 lines) is not loaded by default to keep the page fast.Load diff
added
paper/tables/ops.tex
+14 −0
Large diff (14 lines) is not loaded by default to keep the page fast.Load diff
added
paper/uq_logo.jpg
+0 −0
Binary file not shown.
modified
prisme/config.py
+3 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
modified
prisme/controller.py
+11 −5
Large diff (37 lines) is not loaded by default to keep the page fast.Load diff
modified
prisme/flops.py
+9 −2
Large diff (27 lines) is not loaded by default to keep the page fast.Load diff
modified
prisme/generation.py
+39 −2
Large diff (65 lines) is not loaded by default to keep the page fast.Load diff
modified
prisme/model.py
+10 −2
Large diff (30 lines) is not loaded by default to keep the page fast.Load diff
modified
prisme/readout.py
+1 −1
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added
reproducibility/environment.md
+12 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
added
results/analyzed/labels_labels_p4_seed01_20261030000543.json
+236 −0
Large diff (237 lines) is not loaded by default to keep the page fast.Load diff
added
results/claim_registry.csv
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
results/figures/labels_analysis.pdf
+0 −0
Binary file not shown.
added
results/figures/labels_analysis.png
+0 −0
Binary file not shown.
added
results/figures/learnability_bootstrap.pdf
+0 −0
Binary file not shown.
added
results/figures/learnability_bootstrap.png
+0 −0
Binary file not shown.
added
results/raw/latency_p4_editor_v1_Ustd02_seed01_20261029235050_mps.json
+302 −0
Large diff (303 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/launch_103m.sh
+20 −0
Large diff (20 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/launch_seed_matrix.sh
+35 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/make_results_tables.py
+174 −0
Large diff (174 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/make_tables.py
+120 −0
Large diff (120 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/md2tex.py
+27 −0
Large diff (27 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/node_logs.sh
+6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/plotstyle.py
+28 −0
Large diff (28 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/reproduce_figures.py
+186 −0
Large diff (186 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/verify_manuscript_claims.py
+53 −0
Large diff (53 lines) is not loaded by default to keep the page fast.Load diff
modified
tests/unit/test_invariants.py
+17 −0
Large diff (20 lines) is not loaded by default to keep the page fast.Load diff
modified
training/collect_interventions.py
+1 −1
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
modified
training/common.py
+7 −1
Large diff (14 lines) is not loaded by default to keep the page fast.Load diff
modified
training/train_controller.py
+14 −3
Large diff (41 lines) is not loaded by default to keep the page fast.Load diff
modified
training/train_editor.py
+8 −2
Large diff (27 lines) is not loaded by default to keep the page fast.Load diff