SPB Git
spboucher.ai

spb/wp31_uqo

Public

WP31 PRISME: phases 0-3 implementation, tests, data pipeline, training scripts

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Oct 30, 2026) Browse files

30 changed files +20,001 −0

30 changed files
Load all 30 files Unified Split
added .gitignore +24 −0
@@ -0,0 +1,24 @@
1+.venv/
2+__pycache__/
3+*.pyc
4+.pytest_cache/
5+.hypothesis/
6+experiments/runs/**/*.pt
7+experiments/runs/**/*.safetensors
8+experiments/runs/**/*.bin
9+data/raw/
10+data/cache/
11+data/processed/*.bin
12+data/processed/*.npy
13+*.log.gz
14+paper/*.aux
15+paper/*.bbl
16+paper/*.blg
17+paper/*.fls
18+paper/*.fdb_latexmk
19+paper/*.out
20+paper/*.toc
21+paper/*.synctex.gz
22+paper/*.lof
23+paper/*.lot
24+.DS_Store
added CLAUDE.md +15 −0
@@ -0,0 +1,15 @@
1+# CLAUDE.md — WP31: PRISME-SLM (execution mandate)
2+
3+This repository executes UQO Working Paper 31, *PRISME: Intervention-Supervised Sparse Workspace Editing for
4+Small Language Models* (author: Simon-Pierre Boucher, UQO). The full research mandate (phases 0–15, immutable
5+rules, hypotheses H1–H7, baselines, ablations, paper structure) was supplied by the author on 2026-10-29 and is
6+reproduced in `docs/MANDATE.md`. Working rules that bind every session in this repository:
7+
8+1. Scientific truth first: never fabricate runs, metrics, citations, hardware or proofs. Every number in the
9+ manuscript traces to a run id in `experiments/runs/` and `results/claim_registry.csv`.
10+2. Phases are executed in order with acceptance gates logged in `PROGRESS.md`; failures go to `FAILURES.md`,
11+ design changes to `DECISIONS.md`. Unfavourable runs are never deleted.
12+3. Minimal falsifiable version first (no persistent memory, no RAG, no MoE, no RL, no tool use).
13+4. Resource boundaries: laptop (M5 Max) + existing MacLustr SSH access only; no purchases, no new credentials.
14+5. Independent train / calibration / dev / locked-test splits; test splits are not read before
15+ `protocol/preregistration.md` is frozen (see `protocol/locked_test_manifest.json`).
added DECISIONS.md +13 −0
@@ -0,0 +1,13 @@
1+# DECISIONS.md — design decisions (dated)
2+
3+| id | date | decision | rationale |
4+|---|---|---|---|
5+| D1 | 2026-10-29 | Repository lives at `Recherche-UQO/ia/prisme-slm` (new `ia/` category of the UQO series); Python 3.12 venv via `uv`, PyTorch 2.15 (CPU/MPS), fp32 training. | Series convention; bf16 autocast gave no speed-up on MPS (15.3k vs 15.0k tok/s on M3 Ultra). |
6+| D2 | 2026-10-29 | A dedicated `bank_norm` RMSNorm precedes the bank/z projections. | The pre-final-norm residual stream has growing norm; projecting it raw would couple editor dynamics to residual scale. Costs d parameters. |
7+| D3 | 2026-10-29 | Workspace/controller parameter counts are NOT forced to the brief's figures. | The brief's figures are reproduced exactly by a closed form (`4dw+Mw+17w²+6wf_w+15.5w+4`); the implemented design needs 7 fewer w×w projections per config (bank is read directly; readout uses raw slot values; controller shares one state MLP across the 24 actions so that scoring costs O(A·w²) not O(A·w·f_w)). Reconciliation reported in `results/tables/param_counts.json`. |
8+| D4 | 2026-10-29 | STOP is absorbing in forced-action training too. | Needed for incremental/full-prefix equivalence (test revealed the mismatch). |
9+| D5 | 2026-10-29 | Cost proxy c(a) in the controller's Q is the analytic ops ratio of the action (not a learned ĉ); the ĉ head is kept as a diagnostic regressor. | Per-action cost is deterministic given k(a) and T; learning it adds noise without information. |
10+| D6 | 2026-10-29 | Synthetic splits are deduplicated by exact text across splits (train written first); eval items colliding with train are dropped and counted. | Low-depth cells have few distinct items; 995 train dups, 256 dev_iid dups removed. |
11+| D7 | 2026-10-29 | Phase 4 uses a character-level tokenizer and a 2.1M-parameter backbone trained from scratch on the synthetic train split. | Self-contained learnability test independent of the web corpus/tokenizer. |
12+| D8 | 2026-10-29 | 103M pilot backbone: 196.6M tokens (12k steps × 32 × 512) of FineWeb-Edu, one seed; parameter-matched plain decoder baseline (ffn 2144, 102.16M) trained with the identical recipe on a second node. | ~3.7 h per run at 14.8k tok/s on an M3 Ultra; seeds are varied at the editor/controller stage where the pilot's decisive comparison lives. |
13+| D9 | 2026-10-29 | `readout_scale`: Δ = readout_scale · U(r(S_f) − r(S0)), with readout_scale a fixed constant set once at bootstrap to the measured rms of the backbone residual stream (5.6 for the tiny backbone). | Scale mismatch blocked the U = 0 bootstrap (see FAILURES). The final RMSNorm makes logits invariant to the joint scale of h and Δ, so this is a reparametrisation of U's learning rate; identity properties are unchanged (tests pass). |
added FAILURES.md +7 −0
@@ -0,0 +1,7 @@
1+# FAILURES.md — failed hypotheses, tests and attempts (never deleted)
2+
3+| date | what failed | evidence | resolution |
4+|---|---|---|---|
5+| 2026-10-29 | `test_incremental_equals_full_prefix_forced`: cached decoding ≠ full-prefix forward. | pytest, atol 1e-4 | Root cause: forced-action forward let edits resume after STOP; made STOP absorbing (D4). |
6+| 2026-10-29 | rsync exclusion `experiments` also dropped `configs/experiments/`; baseline launch on M3U96b failed twice (`FileNotFoundError`). | `experiments/logs/pretrain_enlarged102m_seed1.log` (first attempts) | anchored pattern `/experiments`. |
7+| 2026-10-29 | Stage B bootstrap degeneracy (brief §13.1 confirmed): with U = 0 and lr 1e-3, 700 steps of forced mixed-budget training left val loss identical at every budget (0.4144 / 0.4143 / 0.4143 / 0.4146 / 0.4164 for budgets 0/1/2/4/8); grad-norm 0.01–0.02. Diagnosis: scale mismatch — backbone residual rms 5.64 vs readout difference rms 0.06, so U must grow by ×100 before edits can affect logits. | `experiments/runs/p4_editor_v1_seed01_*/metrics.jsonl` (stopped at step ~750) | D9: fixed non-learned `readout_scale` = measured backbone residual rms at bootstrap (keeps U = 0 init and exact identity; equivalent to a scale-aware learning rate). Variants with nonzero U init (std 0.02) and lr 3e-3 kept running on M2U64 / M4M64c as evidence. |
added Makefile +10 −0
@@ -0,0 +1,10 @@
1+PY=.venv/bin/python
2+.PHONY: env test audit params flops paper figures smoke
3+env: ; uv venv --python 3.12 .venv && uv pip install -p .venv/bin/python -r requirements-lock.txt
4+test: ; $(PY) -m pytest tests -q
5+audit: ; $(PY) scripts/environment_audit.py --smoke
6+params: ; $(PY) scripts/count_params.py --all-configs --device meta
7+flops: ; $(PY) scripts/estimate_flops.py --config configs/prisme_459m.yaml --T 4096
8+smoke: ; $(PY) -m training.pretrain --config configs/experiments/pretrain_103m.yaml --dry-run
9+figures: ; $(PY) scripts/reproduce_figures.py --results results/raw/
10+paper: ; cd paper && latexmk -pdf -interaction=nonstopmode main.tex
added PROGRESS.md +10 −0
@@ -0,0 +1,10 @@
1+# PROGRESS.md — WP31 PRISME phase log
2+
3+| date | phase | status | commands / evidence | measurements | gate | blockers / next |
4+|---|---|---|---|---|---|---|
5+| 2026-10-29 | 0 env audit | done | `scripts/environment_audit.py --smoke` → `hardware/device_inventory.json`; `scripts/cluster_audit.sh` (read-only SSH) | laptop M5 Max 18c/40 GPU/48 GB, torch 2.15 MPS+CPU smoke OK; 12 LAN nodes reachable incl. 2× M3 Ultra 96 GB, M2 Ultra 64 GB, 3× M4 Max 64 GB | PASS | — |
6+| 2026-10-29 | 1 literature | done | `literature/papers.bib` (77 verified entries), `comparison_matrix.csv`, `novelty_audit.md` | all 12 mandated refs exist incl. RecurTrace 2609.03379 and ELR 2610.12582; C3 (differential zero-init bypass) judged not novel; C2 novel only in its multi-action+propensity form | PASS | preregistration frozen before any test split is read (see protocol/) |
7+| 2026-10-29 | 2 backbone + counts | done | `scripts/count_params.py --all-configs --device meta`; `scripts/estimate_flops.py` | backbone counts match brief exactly (99,502,848 / 446,744,064 / 1,149,339,648); workspace 2,631,173 vs brief 3,088,260 (reconciled, D3); ops table reproduces brief within 1.3 % | PASS | — |
8+| 2026-10-29 | 3 invariants | done | `python -m pytest tests -q` → 22 passed | 15 mandated invariants + controller target-freedom + dense ablation + 5 cache-equivalence tests | PASS | — |
9+| 2026-10-29 | 4 learnability | running | `training.train_editor --config configs/experiments/p4_tiny_backbone.yaml --device mps` (laptop) | — | — | then editor bootstrap on frozen tiny backbone |
10+| 2026-10-29 | 8 pilot (Stage A) | running | M3U96a: `pretrain_103m.yaml` (run `pretrain103m_seed01_*`); M3U96b: `pretrain_enlarged_102m.yaml` | 14.8k tok/s, 196.6M tokens planned (~3.7 h each) | — | editor/controller stages after completion |
added baselines/__init__.py +0 −0
added baselines/global_recurrence.py +78 −0
@@ -0,0 +1,78 @@
1+"""Global latent-recurrence adapter (Huginn/Ouro/ELR-*inspired*, simplified): the whole top representation is
2+refined r times by ONE shared block operating at workspace width w, with a zero-initialised residual back to d.
3+
4+Same frozen backbone, same forced mixed-budget training recipe and same budget semantics as PRISME; the only
5+difference is that every "action" rewrites the entire latent (global, dense) instead of 1-2 of 16 slots, and
6+there is no slot selection to learn. Disclosed simplifications: no prelude/coda retraining, no truncated
7+backprop through time, adapter-level recurrence on a frozen backbone rather than from-scratch recurrent pretraining.
8+Parameter count is matched to the PRISME workspace (ffn width chosen in the experiment config)."""
9+from __future__ import annotations
10+
11+import torch
12+import torch.nn as nn
13+
14+from prisme.attention import RMSNorm, SwiGLU
15+from prisme.backbone import Backbone
16+from prisme.config import PrismeConfig
17+import torch.nn.functional as F
18+
19+
20+class RecurrentBlock(nn.Module):
21+ def __init__(self, w: int, heads: int, ffn: int, eps: float):
22+ super().__init__()
23+ self.h, self.hd = heads, w // heads
24+ self.n1 = RMSNorm(w, eps); self.q = nn.Linear(w, w, bias=False); self.k = nn.Linear(w, w, bias=False)
25+ self.v = nn.Linear(w, w, bias=False); self.o = nn.Linear(w, w, bias=False)
26+ self.n2 = RMSNorm(w, eps); self.ffn = SwiGLU(w, ffn)
27+
28+ def forward(self, x, ctx_mask):
29+ B, T, w = x.shape
30+ xn = self.n1(x)
31+ q = self.q(xn).view(B, T, self.h, self.hd).transpose(1, 2)
32+ k = self.k(xn).view(B, T, self.h, self.hd).transpose(1, 2)
33+ v = self.v(xn).view(B, T, self.h, self.hd).transpose(1, 2)
34+ a = F.scaled_dot_product_attention(q, k, v, attn_mask=ctx_mask)
35+ x = x + self.o(a.transpose(1, 2).reshape(B, T, w))
36+ return x + self.ffn(self.n2(x))
37+
38+
39+class GlobalRecurrenceModel(nn.Module):
40+ """Interface-compatible with PrismeModel.forward(tokens, actions): any nonterminal action = one global
41+ refinement step; STOP = no step. n_edits semantics identical."""
42+
43+ def __init__(self, cfg: PrismeConfig, ffn_w: int | None = None, heads: int | None = None):
44+ super().__init__()
45+ self.cfg = cfg
46+ self.backbone = Backbone(cfg)
47+ w = cfg.w
48+ self.bank_norm = RMSNorm(cfg.d_model, cfg.norm_eps)
49+ self.down = nn.Linear(cfg.d_model, w, bias=False)
50+ self.block = RecurrentBlock(w, heads or cfg.n_editor_heads, ffn_w or cfg.ffn_w, cfg.norm_eps)
51+ self.ctx_proj = nn.Linear(2, w, bias=True)
52+ self.up = nn.Linear(w, cfg.d_model, bias=False)
53+ nn.init.zeros_(self.up.weight); nn.init.zeros_(self.ctx_proj.weight); nn.init.zeros_(self.ctx_proj.bias)
54+ self.controller = nn.Identity() # placeholder so parameter-group helpers work
55+
56+ def forward(self, tokens, actions=None, return_state=False):
57+ cfg = self.cfg
58+ H = self.backbone(tokens)
59+ if actions is None or actions.shape[-1] == 0:
60+ return {"logits": self.backbone.head(H), "H": H, "n_edits": None}
61+ B, T, R = actions.shape
62+ x0 = self.down(self.bank_norm(H))
63+ x = x0
64+ mask = torch.ones(T, T, dtype=torch.bool, device=H.device).tril().view(1, 1, T, T)
65+ stopped = torch.zeros(B, T, dtype=torch.bool, device=H.device)
66+ n_edits = torch.zeros(B, T, dtype=torch.long, device=H.device)
67+ for r in range(R):
68+ a = torch.where(stopped, torch.full_like(actions[:, :, r], cfg.stop_action), actions[:, :, r])
69+ act = (a != cfg.stop_action)
70+ stopped |= ~act
71+ ctx = torch.stack([torch.full((B, T), r / cfg.r_max, device=H.device), torch.full((B, T), (R - r - 1) / cfg.r_max, device=H.device)], -1)
72+ xn = self.block(x + self.ctx_proj(ctx), mask)
73+ x = torch.where(act.unsqueeze(-1), xn, x) # positions that stopped keep their state
74+ n_edits += act.long()
75+ delta = self.up(x - x0)
76+ edited = n_edits > 0
77+ delta = torch.where(edited.unsqueeze(-1), delta, torch.zeros_like(delta))
78+ return {"logits": self.backbone.head(H + delta), "H": H, "n_edits": n_edits, "delta_out": delta}
added baselines/routers.py +54 −0
@@ -0,0 +1,54 @@
1+"""Reference routing policies that share PRISME's editor (actions chosen without the learned value controller).
2+All are target-free and deployable; they differ only in how the action is picked at each step.
3+ random uniform over the 24 nonterminal actions, fixed budget b
4+ uniform round-robin single-slot schedule SINGLE_0, SINGLE_1, ... (deterministic, fixed budget b)
5+ fixed_pair PAIR_0_1 repeated (fixed budget b)
6+ entropy continue (random action) while the current predictive entropy exceeds a threshold, else STOP
7+ norm continue while the last write moved the slot by more than a threshold (RMS), else STOP
8+"""
9+from __future__ import annotations
10+
11+import torch
12+
13+from prisme.model import PrismeModel
14+
15+
16+@torch.no_grad()
17+def rollout(model: PrismeModel, tokens, policy: str, budget: int, threshold: float = 0.0, seed: int = 0):
18+ cfg = model.cfg
19+ ctx = model.encode(tokens)
20+ S = ctx.S0
21+ B, T = tokens.shape
22+ g = torch.Generator(device="cpu").manual_seed(seed)
23+ stopped = torch.zeros(B, T, dtype=torch.bool, device=tokens.device)
24+ n_edits = torch.zeros(B, T, dtype=torch.long, device=tokens.device)
25+ trace = torch.full((B, T, budget), cfg.stop_action, dtype=torch.long, device=tokens.device)
26+ last_rms = torch.full((B, T), float("inf"), device=tokens.device)
27+ for r in range(budget):
28+ if policy == "random":
29+ a = torch.randint(0, cfg.n_actions, (B, T), generator=g).to(tokens.device)
30+ elif policy == "uniform":
31+ a = torch.full((B, T), r % cfg.n_slots, device=tokens.device)
32+ elif policy == "fixed_pair":
33+ a = torch.full((B, T), cfg.n_slots + (r % cfg.n_pairs), device=tokens.device)
34+ elif policy == "entropy":
35+ logits, _ = model.logits_from_state(S, ctx, n_edits > 0)
36+ p = torch.softmax(logits.float(), -1)
37+ ent = -(p * torch.log(p.clamp_min(1e-12))).sum(-1)
38+ a = torch.randint(0, cfg.n_actions, (B, T), generator=g).to(tokens.device)
39+ a = torch.where(ent > threshold, a, torch.full_like(a, cfg.stop_action))
40+ elif policy == "norm":
41+ a = torch.randint(0, cfg.n_actions, (B, T), generator=g).to(tokens.device)
42+ a = torch.where(last_rms > threshold, a, torch.full_like(a, cfg.stop_action))
43+ else:
44+ raise ValueError(policy)
45+ a = torch.where(stopped, torch.full_like(a, cfg.stop_action), a)
46+ stopped |= a == cfg.stop_action
47+ trace[:, :, r] = a
48+ S_new, info = model.apply_action(S, ctx, a, r, budget - r - 1)
49+ moved = (info["eta"].squeeze(-1) * info["delta_rms"] * info["scale"]) # [B,T,k] realised rms step per lane
50+ last_rms = torch.where(a != cfg.stop_action, moved.max(-1).values, last_rms)
51+ S = S_new
52+ n_edits += (a != cfg.stop_action).long()
53+ logits, _ = model.logits_from_state(S, ctx, n_edits > 0)
54+ return {"logits": logits, "actions": trace, "n_edits": n_edits}
added configs/baselines/transformer_enlarged_102m.yaml +23 −0
@@ -0,0 +1,23 @@
1+# PRISME reference configuration: prisme_103m (see docs/architecture_spec.md)
2+model:
3+ name: transformer_enlarged_102m
4+ vocab_size: 32768
5+ d_model: 768
6+ n_layers: 12
7+ n_heads: 12
8+ n_kv_heads: 3
9+ ffn_width: 2144 # parameter-matched plain decoder: +2.65M params ~= PRISME workspace+controller (2.63M)
10+ max_seq_len: 4096
11+ rope_theta: 10000.0
12+ norm_eps: 1.0e-5
13+ tie_embeddings: true
14+ use_workspace: false
15+ w: 256
16+ ffn_w: 768
17+ n_slots: 16
18+ n_editor_blocks: 2
19+ n_editor_heads: 4
20+ r_max: 8
21+ eta_max: 0.25
22+ clip_c: 1.0
23+ readout_zero_init: true
added configs/experiments/controller_p4.yaml +14 −0
@@ -0,0 +1,14 @@
1+# Phase 6 (tiny scale): value controller fit on logged labels; (lambda, kappa) tuned on calibration rollouts.
2+name: controller_p4
3+seed: 1
4+mode: prisme
5+model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512,
6+ max_seq_len: 320, w: 64, ffn_w: 192, n_slots: 16, n_editor_heads: 2, r_max: 8}
7+checkpoint: CHECKPOINT_DIR
8+labels: {train: LABELS_TRAIN, calib: LABELS_CALIB}
9+train: {steps: 3000, lr: 1.0e-3, weight_decay: 0.01, alpha: 1.0, beta: 0.5, gamma: 0.1, huber_delta: 0.1, log_every: 100, eval_every: 1000, seq_len: 320}
10+lambda_grid: [0.0, 0.01, 0.03, 0.1, 0.3]
11+kappa_grid: [0.0, 0.5]
12+sweep_batches: 10
13+max_budget: 8
14+selection_rule: max_loss_reduction_per_ops
added configs/experiments/labels_p4.yaml +13 −0
@@ -0,0 +1,13 @@
1+# Phase 5 (tiny scale): counterfactual label collection with the fixed Phase 4 editor checkpoint.
2+name: labels_p4
3+seed: 1
4+mode: prisme
5+model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512,
6+ max_seq_len: 320, w: 64, ffn_w: 192, n_slots: 16, n_editor_heads: 2, r_max: 8}
7+checkpoint: CHECKPOINT_DIR
8+use_policy: false
9+data: {kind: synthetic, tokenizer: char, splits_dir: data/splits/synthetic_v1, loss_span: answer}
10+collect: {batch_size: 32, seq_len: 320, max_prefix: 3, audit_every: 4}
11+partitions:
12+ train: {split: train, n_batches: 120, stream_seed: 11}
13+ calib: {split: calib, n_batches: 30, stream_seed: 12}
added configs/experiments/p4_editor_bootstrap.yaml +16 −0
@@ -0,0 +1,16 @@
1+# Phase 4 step 2: Stage B editor bootstrap on the FROZEN tiny backbone (v1), forced mixed budgets {0,1,2,4,8}.
2+name: p4_editor_v1_scaled
3+seed: 1
4+mode: prisme
5+model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512,
6+ max_seq_len: 320, w: 64, ffn_w: 192, n_slots: 16, n_editor_heads: 2, r_max: 8}
7+init: {backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint, freeze_backbone: true, auto_readout_scale: true}
8+data: {kind: synthetic, tokenizer: char, splits_dir: data/splits/synthetic_v1, loss_span: answer,
9+ eval_splits: [dev_iid, dev_template, dev_depth, dev_compose]}
10+budgets: [0, 1, 2, 4, 8]
11+budget_probs: [0.2, 0.2, 0.2, 0.2, 0.2]
12+budget_per: position
13+train: {batch_size: 32, seq_len: 320, steps: 3000, lr: 1.0e-3, warmup: 100, weight_decay: 0.1, grad_clip: 1.0,
14+ log_every: 50, eval_every: 500, eval_batches: 8, ckpt_every: 1500}
15+em_budgets: [0, 2, 8]
16+em_max_items: 400
added configs/experiments/p4_editor_bootstrap_lr3e3.yaml +16 −0
@@ -0,0 +1,16 @@
1+# Phase 4 step 2: Stage B editor bootstrap on the FROZEN tiny backbone (v1), forced mixed budgets {0,1,2,4,8}.
2+name: p4_editor_v1_lr3e3_b64
3+seed: 1
4+mode: prisme
5+model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512,
6+ max_seq_len: 320, w: 64, ffn_w: 192, n_slots: 16, n_editor_heads: 2, r_max: 8}
7+init: {backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint, freeze_backbone: true}
8+data: {kind: synthetic, tokenizer: char, splits_dir: data/splits/synthetic_v1, loss_span: answer,
9+ eval_splits: [dev_iid, dev_template, dev_depth, dev_compose]}
10+budgets: [0, 1, 2, 4, 8]
11+budget_probs: [0.2, 0.2, 0.2, 0.2, 0.2]
12+budget_per: position
13+train: {batch_size: 64, seq_len: 320, steps: 2000, lr: 3.0e-3, warmup: 100, weight_decay: 0.1, grad_clip: 1.0,
14+ log_every: 50, eval_every: 500, eval_batches: 8, ckpt_every: 1500}
15+em_budgets: [0, 2, 8]
16+em_max_items: 400
added configs/experiments/p4_editor_bootstrap_nonzeroU.yaml +17 −0
@@ -0,0 +1,17 @@
1+# Phase 4 step 2: Stage B editor bootstrap on the FROZEN tiny backbone (v1), forced mixed budgets {0,1,2,4,8}.
2+name: p4_editor_v1_nonzeroU
3+seed: 1
4+mode: prisme
5+model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512,
6+ max_seq_len: 320, w: 64, ffn_w: 192, n_slots: 16, n_editor_heads: 2, r_max: 8}
7+init: {backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint, freeze_backbone: true}
8+data: {kind: synthetic, tokenizer: char, splits_dir: data/splits/synthetic_v1, loss_span: answer,
9+ eval_splits: [dev_iid, dev_template, dev_depth, dev_compose]}
10+budgets: [0, 1, 2, 4, 8]
11+budget_probs: [0.2, 0.2, 0.2, 0.2, 0.2]
12+budget_per: position
13+train: {batch_size: 32, seq_len: 320, steps: 2000, lr: 1.0e-3, warmup: 100, weight_decay: 0.1, grad_clip: 1.0,
14+ log_every: 50, eval_every: 500, eval_batches: 8, ckpt_every: 1500}
15+em_budgets: [0, 2, 8]
16+em_max_items: 400
17+model_overrides: {readout_zero_init: false}
added configs/experiments/p4_tiny_backbone.yaml +12 −0
@@ -0,0 +1,12 @@
1+# Phase 4 step 1: tiny backbone pretrained from scratch on the synthetic TRAIN split (char-level), workspace off.
2+name: p4_tiny_backbone
3+seed: 1
4+mode: backbone_only
5+model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512,
6+ max_seq_len: 320, w: 64, ffn_w: 192, n_slots: 16, n_editor_heads: 2, r_max: 8}
7+init: {freeze_backbone: false}
8+data: {kind: synthetic, tokenizer: char, splits_dir: data/splits/synthetic_v1, loss_span: all,
9+ eval_splits: [dev_iid, dev_template, dev_depth, dev_compose]}
10+train: {batch_size: 64, seq_len: 320, steps: 6000, lr: 1.0e-3, warmup: 200, weight_decay: 0.1, grad_clip: 1.0,
11+ log_every: 50, eval_every: 500, eval_batches: 10, ckpt_every: 2000}
12+em_max_items: 600
added configs/experiments/p4_tiny_backbone_v2.yaml +12 −0
@@ -0,0 +1,12 @@
1+# Phase 4 step 1 (v2): stronger tiny backbone (≈5.6M params) from scratch on the synthetic TRAIN split, char-level.
2+name: p4_tiny_backbone_v2
3+seed: 1
4+mode: backbone_only
5+model: {name: tiny_p4v2, vocab_size: 128, d_model: 256, n_layers: 6, n_heads: 8, n_kv_heads: 4, ffn_width: 768,
6+ max_seq_len: 320, w: 64, ffn_w: 192, n_slots: 16, n_editor_heads: 2, r_max: 8}
7+init: {freeze_backbone: false}
8+data: {kind: synthetic, tokenizer: char, splits_dir: data/splits/synthetic_v1, loss_span: all,
9+ eval_splits: [dev_iid, dev_template, dev_depth, dev_compose]}
10+train: {batch_size: 64, seq_len: 320, steps: 15000, lr: 1.0e-3, warmup: 300, weight_decay: 0.1, grad_clip: 1.0,
11+ log_every: 100, eval_every: 1000, eval_batches: 10, ckpt_every: 5000}
12+em_max_items: 800
added configs/experiments/pretrain_103m.yaml +18 −0
@@ -0,0 +1,18 @@
1+name: pretrain103m
2+seed: 1
3+model: configs/prisme_103m.yaml
4+data:
5+ train_bin: data/processed/fineweb_edu_300m/train.bin
6+ val_bin: data/processed/fineweb_edu_300m/val.bin
7+train:
8+ batch_size: 32
9+ seq_len: 512
10+ steps: 12000 # 32*512*12000 = 196.6M tokens (pilot budget; see protocol/preregistration.md)
11+ lr: 6.0e-4
12+ warmup: 300
13+ weight_decay: 0.1
14+ grad_clip: 1.0
15+ log_every: 20
16+ eval_every: 500
17+ eval_batches: 20
18+ ckpt_every: 1000
added configs/experiments/pretrain_enlarged_102m.yaml +18 −0
@@ -0,0 +1,18 @@
1+name: pretrain_enlarged102m
2+seed: 1
3+model: configs/baselines/transformer_enlarged_102m.yaml
4+data:
5+ train_bin: data/processed/fineweb_edu_300m/train.bin
6+ val_bin: data/processed/fineweb_edu_300m/val.bin
7+train:
8+ batch_size: 32
9+ seq_len: 512
10+ steps: 12000 # 32*512*12000 = 196.6M tokens (pilot budget; see protocol/preregistration.md)
11+ lr: 6.0e-4
12+ warmup: 300
13+ weight_decay: 0.1
14+ grad_clip: 1.0
15+ log_every: 20
16+ eval_every: 500
17+ eval_batches: 20
18+ ckpt_every: 1000
added configs/experiments/routing_p4.yaml +13 −0
@@ -0,0 +1,13 @@
1+# Routing comparison (tiny scale) on dev splits with the LOCKED controller checkpoint.
2+seed: 1
3+mode: prisme
4+model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512,
5+ max_seq_len: 320, w: 64, ffn_w: 192, n_slots: 16, n_editor_heads: 2, r_max: 8}
6+checkpoint: CHECKPOINT_DIR
7+data: {kind: synthetic, tokenizer: char, splits_dir: data/splits/synthetic_v1, loss_span: answer, split: dev_iid}
8+batch_size: 32
9+seq_len: 320
10+n_batches: 20
11+budgets: [1, 2, 4, 8]
12+oracle_batches: 4
13+two_step_batches: 1
added configs/prisme_103m.yaml +23 −0
@@ -0,0 +1,23 @@
1+# PRISME reference configuration: prisme_103m (see docs/architecture_spec.md)
2+model:
3+ name: prisme_103m
4+ vocab_size: 32768
5+ d_model: 768
6+ n_layers: 12
7+ n_heads: 12
8+ n_kv_heads: 3
9+ ffn_width: 2048
10+ max_seq_len: 4096
11+ rope_theta: 10000.0
12+ norm_eps: 1.0e-5
13+ tie_embeddings: true
14+ use_workspace: true
15+ w: 256
16+ ffn_w: 768
17+ n_slots: 16
18+ n_editor_blocks: 2
19+ n_editor_heads: 4
20+ r_max: 8
21+ eta_max: 0.25
22+ clip_c: 1.0
23+ readout_zero_init: true
added configs/prisme_1180m.yaml +23 −0
@@ -0,0 +1,23 @@
1+# PRISME reference configuration: prisme_1180m (see docs/architecture_spec.md)
2+model:
3+ name: prisme_1180m
4+ vocab_size: 32768
5+ d_model: 2048
6+ n_layers: 24
7+ n_heads: 32
8+ n_kv_heads: 8
9+ ffn_width: 5632
10+ max_seq_len: 4096
11+ rope_theta: 10000.0
12+ norm_eps: 1.0e-5
13+ tie_embeddings: true
14+ use_workspace: true
15+ w: 768
16+ ffn_w: 2048
17+ n_slots: 16
18+ n_editor_blocks: 2
19+ n_editor_heads: 12
20+ r_max: 8
21+ eta_max: 0.25
22+ clip_c: 1.0
23+ readout_zero_init: true
added configs/prisme_459m.yaml +23 −0
@@ -0,0 +1,23 @@
1+# PRISME reference configuration: prisme_459m (see docs/architecture_spec.md)
2+model:
3+ name: prisme_459m
4+ vocab_size: 32768
5+ d_model: 1536
6+ n_layers: 16
7+ n_heads: 24
8+ n_kv_heads: 6
9+ ffn_width: 4096
10+ max_seq_len: 4096
11+ rope_theta: 10000.0
12+ norm_eps: 1.0e-5
13+ tie_embeddings: true
14+ use_workspace: true
15+ w: 512
16+ ffn_w: 1536
17+ n_slots: 16
18+ n_editor_blocks: 2
19+ n_editor_heads: 8
20+ r_max: 8
21+ eta_max: 0.25
22+ clip_c: 1.0
23+ readout_zero_init: true
added data/__init__.py +0 −0
added data/char_tokenizer.py +29 −0
@@ -0,0 +1,29 @@
1+"""Self-contained character-level tokenizer for the Phase 4 learnability experiments (no corpus dependency)."""
2+from __future__ import annotations
3+
4+import json
5+from pathlib import Path
6+
7+PRINTABLE = [chr(i) for i in range(32, 127)]
8+
9+
10+class CharTokenizer:
11+ def __init__(self):
12+ self.specials = ["<eot>", "<pad>"]
13+ self.itos = self.specials + PRINTABLE
14+ self.stoi = {s: i for i, s in enumerate(self.itos)}
15+ self.eot_id = 0
16+ self.pad_id = 1
17+
18+ @property
19+ def vocab_size(self) -> int:
20+ return len(self.itos)
21+
22+ def encode(self, text: str) -> list[int]:
23+ return [self.stoi.get(ch, self.stoi["?"]) for ch in text]
24+
25+ def decode(self, ids) -> str:
26+ return "".join(self.itos[i] if i >= len(self.specials) else "" for i in ids)
27+
28+ def save(self, path):
29+ Path(path).write_text(json.dumps({"itos": self.itos}))
added data/generators/__init__.py +0 −0
added data/generators/synthetic.py +236 −0
@@ -0,0 +1,236 @@
1+"""Verifiable synthetic task generators with programmatic ground truth.
2+
3+Families (each item is a dict: family, template, depth, prompt, answer, text, meta):
4+ arith multi-step arithmetic chains (depth = number of operations)
5+ bind variable binding + dependent substitution (depth = chain length)
6+ fsm register / finite-state updates (depth = number of updates)
7+ excl conjunctive constraints with distractors (depth = number of constraints)
8+ sched rule-based selection under a budget and exclusions (depth = number of candidates)
9+ compose held-out composition: bind chain feeding an arithmetic expression (depth = total steps)
10+
11+Splitting is done BEFORE sampling: by (family, template id, depth regime, seed stream). Train uses templates
12+T0-T2 and depths <= max_train_depth; held-out "template" items use templates T3-T4; held-out "depth" items use
13+depths > max_train_depth; the compose family is entirely held out from training unless explicitly requested.
14+Text format: "<prompt> Answer: <answer>" ; the answer is the only supervised span for exact-match evaluation.
15+"""
16+from __future__ import annotations
17+
18+import hashlib
19+import json
20+import random
21+from dataclasses import dataclass, asdict
22+from typing import Callable
23+
24+OPS = {"+": lambda a, b: a + b, "-": lambda a, b: a - b, "*": lambda a, b: a * b}
25+
26+# Phrasing templates: index 0..2 = train, 3..4 = held-out paraphrases.
27+TEMPLATES = {
28+ "arith": ["Compute {expr}.", "What is {expr}?", "Evaluate the expression {expr}.", "Calculate {expr} step by step and give the result.", "Find the value of {expr}."],
29+ "bind": ["Let {chain}. What is {var}?", "Given {chain}, compute {var}.", "{chain}. Find {var}.", "Suppose {chain}. Then {var} equals what?", "With {chain}, the value of {var} is?"],
30+ "fsm": ["Start with register {init}. {ops}. What is the register?", "The counter begins at {init}. {ops}. Final counter?", "Initial value {init}. {ops}. Result?", "A register holds {init}. Apply: {ops}. What does it hold now?", "Begin at {init}; {ops}; final value?"],
31+ "excl": ["Items: {items}. Keep the items that are {rules}. Which items remain?", "Consider {items}. Select those that are {rules}. List them.", "{items}. Filter: {rules}. Remaining items?", "Given the list {items}, keep only the ones {rules}. Answer with the names.", "From {items}, which satisfy: {rules}?"],
32+ "sched": ["Budget {budget}. Tasks: {tasks}. Rule: {rule}. Choose the affordable task with the highest value.", "You have {budget} units. Options: {tasks}. Constraint: {rule}. Which affordable option has the largest value?", "Capacity {budget}. Candidates: {tasks}. Exclusion: {rule}. Pick the best affordable candidate.", "With a budget of {budget}, among {tasks}, under the rule {rule}, which task maximises value while staying affordable?", "Budget is {budget}. {tasks}. {rule}. Best affordable task by value?"],
33+ "compose": ["Let {chain}. Compute {expr}.", "Given {chain}, what is {expr}?", "{chain}. Evaluate {expr}.", "Suppose {chain}. Then find {expr}.", "With {chain}, calculate {expr}."],
34+}
35+
36+
37+def _rng(family: str, template: int, depth: int, seed: int, idx: int) -> random.Random:
38+ key = f"{family}|{template}|{depth}|{seed}|{idx}"
39+ return random.Random(int(hashlib.sha256(key.encode()).hexdigest()[:16], 16))
40+
41+
42+def gen_arith(r: random.Random, depth: int, template: int):
43+ vals = [r.randint(1, 20)]
44+ ops = []
45+ expr = str(vals[0])
46+ cur = vals[0]
47+ for _ in range(depth):
48+ op = r.choice(["+", "-", "*"])
49+ b = r.randint(1, 9) if op == "*" else r.randint(1, 30)
50+ cur = OPS[op](cur, b) # left-to-right evaluation (stated below)
51+ expr += f" {op} {b}"
52+ ops.append(op)
53+ prompt = TEMPLATES["arith"][template].format(expr=expr) + " Evaluate left to right."
54+ return prompt, str(cur), {"ops": "".join(ops)}
55+
56+
57+def gen_bind(r: random.Random, depth: int, template: int):
58+ names = r.sample(list("abcdefghjkmnpqrstuvwxyz"), depth + 1)
59+ vals = {names[0]: r.randint(1, 15)}
60+ parts = [f"{names[0]} = {vals[names[0]]}"]
61+ for i in range(1, depth + 1):
62+ op = r.choice(["+", "-", "*"])
63+ b = r.randint(1, 5) if op == "*" else r.randint(1, 12)
64+ src = names[i - 1]
65+ vals[names[i]] = OPS[op](vals[src], b)
66+ parts.append(f"{names[i]} = {src} {op} {b}")
67+ # distractor: an unused variable
68+ d = r.choice([n for n in "ABCDEFG"])
69+ parts.insert(r.randint(1, len(parts)), f"{d} = {r.randint(1, 40)}")
70+ chain = ", ".join(parts)
71+ target = names[depth]
72+ prompt = TEMPLATES["bind"][template].format(chain=chain, var=target)
73+ return prompt, str(vals[target]), {"target": target}
74+
75+
76+def gen_fsm(r: random.Random, depth: int, template: int):
77+ init = r.randint(0, 10)
78+ cur = init
79+ steps = []
80+ for _ in range(depth):
81+ op = r.choice(["add", "sub", "double", "reset", "halve"])
82+ if op == "add":
83+ b = r.randint(1, 9); cur += b; steps.append(f"add {b}")
84+ elif op == "sub":
85+ b = r.randint(1, 9); cur -= b; steps.append(f"subtract {b}")
86+ elif op == "double":
87+ cur *= 2; steps.append("double it")
88+ elif op == "halve":
89+ cur = cur // 2; steps.append("halve it (round down)")
90+ else:
91+ b = r.randint(0, 9); cur = b; steps.append(f"reset to {b}")
92+ prompt = TEMPLATES["fsm"][template].format(init=init, ops=", then ".join(steps))
93+ return prompt, str(cur), {}
94+
95+
96+def gen_excl(r: random.Random, depth: int, template: int):
97+ n = r.randint(4, 7)
98+ names = r.sample(["apple", "pear", "plum", "fig", "kiwi", "lime", "date", "yam", "nut", "bean"], n)
99+ items = {nm: (r.randint(1, 30), r.choice(["red", "green", "blue"])) for nm in names}
100+ rules = []
101+ def keep(v, c):
102+ return True
103+ preds: list[Callable] = []
104+ for _ in range(depth):
105+ kind = r.choice(["gt", "lt", "color", "even"])
106+ if kind == "gt":
107+ t = r.randint(5, 25); rules.append(f"greater than {t}"); preds.append(lambda v, c, t=t: v > t)
108+ elif kind == "lt":
109+ t = r.randint(5, 25); rules.append(f"smaller than {t}"); preds.append(lambda v, c, t=t: v < t)
110+ elif kind == "color":
111+ col = r.choice(["red", "green", "blue"]); rules.append(f"not {col}"); preds.append(lambda v, c, col=col: c != col)
112+ else:
113+ rules.append("even"); preds.append(lambda v, c: v % 2 == 0)
114+ remaining = [nm for nm in names if all(p(*items[nm]) for p in preds)]
115+ items_s = ", ".join(f"{nm} ({v}, {c})" for nm, (v, c) in items.items())
116+ prompt = TEMPLATES["excl"][template].format(items=items_s, rules=" and ".join(rules))
117+ ans = " ".join(remaining) if remaining else "none"
118+ return prompt, ans, {"n_items": n, "n_remaining": len(remaining)}
119+
120+
121+def gen_sched(r: random.Random, depth: int, template: int):
122+ names = [chr(ord("A") + i) for i in range(depth)]
123+ tasks = {nm: (r.randint(1, 12), r.randint(1, 20)) for nm in names}
124+ budget = r.randint(4, 12)
125+ kind = r.choice(["cost_gt", "value_lt", "none"])
126+ if kind == "cost_gt":
127+ t = r.randint(3, 10); rule = f"exclude tasks with cost greater than {t}"; ok = lambda c, v: c <= t
128+ elif kind == "value_lt":
129+ t = r.randint(3, 10); rule = f"exclude tasks with value below {t}"; ok = lambda c, v: v >= t
130+ else:
131+ rule = "no exclusion"; ok = lambda c, v: True
132+ feas = [(v, nm) for nm, (c, v) in tasks.items() if c <= budget and ok(c, v)]
133+ if not feas:
134+ ans = "none"
135+ else:
136+ best = max(v for v, _ in feas)
137+ ans = sorted(nm for v, nm in feas if v == best)[0]
138+ tasks_s = ", ".join(f"{nm} (cost {c}, value {v})" for nm, (c, v) in tasks.items())
139+ prompt = TEMPLATES["sched"][template].format(budget=budget, tasks=tasks_s, rule=rule)
140+ return prompt, ans, {"n_feasible": len(feas)}
141+
142+
143+def gen_compose(r: random.Random, depth: int, template: int):
144+ # bind chain of length depth-2 feeding a 2-op arithmetic expression over two chain variables
145+ k = max(1, depth - 2)
146+ names = r.sample(list("abcdefghjkmnpqrstuvwxyz"), k + 1)
147+ vals = {names[0]: r.randint(1, 10)}
148+ parts = [f"{names[0]} = {vals[names[0]]}"]
149+ for i in range(1, k + 1):
150+ op = r.choice(["+", "-"]); b = r.randint(1, 9); src = names[i - 1]
151+ vals[names[i]] = OPS[op](vals[src], b); parts.append(f"{names[i]} = {src} {op} {b}")
152+ v1, v2 = r.sample(names, 2) if len(names) > 1 else (names[0], names[0])
153+ op1, op2 = r.choice(["+", "-", "*"]), r.choice(["+", "-"])
154+ c = r.randint(1, 9)
155+ val = OPS[op2](OPS[op1](vals[v1], vals[v2]), c)
156+ expr = f"{v1} {op1} {v2} {op2} {c} (left to right)"
157+ prompt = TEMPLATES["compose"][template].format(chain=", ".join(parts), expr=expr)
158+ return prompt, str(val), {}
159+
160+
161+GENERATORS = {"arith": gen_arith, "bind": gen_bind, "fsm": gen_fsm, "excl": gen_excl, "sched": gen_sched, "compose": gen_compose}
162+
163+
164+def make_item(family: str, template: int, depth: int, seed: int, idx: int) -> dict:
165+ r = _rng(family, template, depth, seed, idx)
166+ prompt, answer, meta = GENERATORS[family](r, depth, template)
167+ text = f"{prompt} Answer: {answer}"
168+ return {"family": family, "template": template, "depth": depth, "seed": seed, "idx": idx,
169+ "prompt": prompt + " Answer:", "answer": " " + answer, "text": text, "meta": meta,
170+ "uid": hashlib.sha256(text.encode()).hexdigest()[:16]}
171+
172+
173+@dataclass
174+class SplitSpec:
175+ name: str
176+ families: tuple[str, ...]
177+ templates: tuple[int, ...]
178+ depths: tuple[int, ...]
179+ seed: int
180+ n_per_cell: int
181+
182+
183+def generate_split(spec: SplitSpec) -> list[dict]:
184+ items = []
185+ for fam in spec.families:
186+ for tpl in spec.templates:
187+ for dep in spec.depths:
188+ for i in range(spec.n_per_cell):
189+ items.append(make_item(fam, tpl, dep, spec.seed, i))
190+ return items
191+
192+
193+def default_splits(n_train_per_cell: int = 400, n_eval_per_cell: int = 60) -> dict[str, SplitSpec]:
194+ """Train: 5 families, templates 0-2, depths 1-4, seed 1. Dev: seed 2 same cells (IID generalisation).
195+ Held-out template: templates 3-4. Held-out depth: depths 5-6. Compose: fully held-out family.
196+ Final test: seed 9 across all eval regimes (LOCKED until preregistration is frozen)."""
197+ fams = ("arith", "bind", "fsm", "excl", "sched")
198+ return {
199+ "train": SplitSpec("train", fams, (0, 1, 2), (1, 2, 3, 4), 1, n_train_per_cell),
200+ "calib": SplitSpec("calib", fams, (0, 1, 2), (1, 2, 3, 4), 3, n_eval_per_cell // 2),
201+ "dev_iid": SplitSpec("dev_iid", fams, (0, 1, 2), (1, 2, 3, 4), 2, n_eval_per_cell),
202+ "dev_template": SplitSpec("dev_template", fams, (3, 4), (1, 2, 3, 4), 2, n_eval_per_cell),
203+ "dev_depth": SplitSpec("dev_depth", fams, (0, 1, 2), (5, 6), 2, n_eval_per_cell),
204+ "dev_compose": SplitSpec("dev_compose", ("compose",), (0, 1, 2), (3, 4, 5), 2, n_eval_per_cell),
205+ "test_iid": SplitSpec("test_iid", fams, (0, 1, 2), (1, 2, 3, 4), 9, n_eval_per_cell),
206+ "test_template": SplitSpec("test_template", fams, (3, 4), (1, 2, 3, 4), 9, n_eval_per_cell),
207+ "test_depth": SplitSpec("test_depth", fams, (0, 1, 2), (5, 6), 9, n_eval_per_cell),
208+ "test_compose": SplitSpec("test_compose", ("compose",), (0, 1, 2), (3, 4, 5), 9, n_eval_per_cell),
209+ }
210+
211+
212+def write_splits(out_dir: str, n_train_per_cell: int = 400, n_eval_per_cell: int = 60) -> dict:
213+ from pathlib import Path
214+ out = Path(out_dir); out.mkdir(parents=True, exist_ok=True)
215+ manifest = {}
216+ all_uids: dict[str, str] = {}
217+ for name, spec in default_splits(n_train_per_cell, n_eval_per_cell).items():
218+ items = generate_split(spec)
219+ # cross-split exact-duplicate removal: an item whose full text already appeared in an earlier split
220+ # (train is written first) is DROPPED from the later split; within-split duplicates are dropped too.
221+ kept, dups = [], 0
222+ for it in items:
223+ if it["uid"] in all_uids:
224+ dups += 1
225+ continue
226+ all_uids[it["uid"]] = name
227+ kept.append(it)
228+ items = kept
229+ p = out / f"{name}.jsonl"
230+ with open(p, "w") as f:
231+ for it in items:
232+ f.write(json.dumps(it) + "\n")
233+ manifest[name] = {"spec": asdict(spec), "n": len(items), "duplicates_dropped": dups,
234+ "sha256": hashlib.sha256(p.read_bytes()).hexdigest(), "locked": name.startswith("test")}
235+ (out / "manifest.json").write_text(json.dumps(manifest, indent=2))
236+ return manifest
added data/processed/fineweb_edu_300m/manifest.json +48 −0
@@ -0,0 +1,48 @@
1+{
2+ "source": "HuggingFaceFW/fineweb-edu sample-10BT (streaming, canonical shard order)",
3+ "license": "ODC-By 1.0 (https://opendatacommons.org/licenses/by/1-0/)",
4+ "prepared_at": "2026-10-29T23:19:53-0400",
5+ "args": {
6+ "out": "data/processed/fineweb_edu_300m",
7+ "target_train_tokens": 300000000,
8+ "vocab_size": 32768,
9+ "tokenizer_docs": 60000,
10+ "val_pct": 0.01,
11+ "test_pct": 0.01,
12+ "max_doc_chars": 40000
13+ },
14+ "docs_streamed": 291647,
15+ "docs": {
16+ "train": 285604,
17+ "val": 2957,
18+ "test": 2973
19+ },
20+ "chars": {
21+ "train": 1290001347,
22+ "val": 12988848,
23+ "test": 12933032
24+ },
25+ "splits": {
26+ "train": {
27+ "tokens": 289959006,
28+ "docs": 285604,
29+ "sha256": "e931e83c4ba6e62a2125e50a5390e8df78913dbed31af805d8af17fafad95123",
30+ "locked": false
31+ },
32+ "val": {
33+ "tokens": 2913824,
34+ "docs": 2957,
35+ "sha256": "0565e3f4a4028b241f40ef07f20bc68b24e45d99e147860be066e3667d1532be",
36+ "locked": false
37+ },
38+ "test": {
39+ "tokens": 2883436,
40+ "docs": 2973,
41+ "sha256": "1ce9ca3ab6d659fe08042844fc0589951e09386ee79d2ef76504ffaaebb0947b",
42+ "locked": true
43+ }
44+ },
45+ "tokenizer_sha256": "5a78df618767945743623d5ff238d035839d02388b4c53c548a437b3d8a206ee",
46+ "eot_id": 0,
47+ "dtype": "uint16"
48+}
\ No newline at end of file
added data/processed/fineweb_edu_300m/test.bin +0 −0

Binary file not shown.

added data/processed/fineweb_edu_300m/tokenizer.json +19236 −0
Large diff (19,236 lines) is not loaded by default to keep the page fast.Load diff

Diff truncated — file too large.

Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere