SPB Git
spboucher.ai

spb/wp31_uqo

Public

WP31 PRISME: phases 0-3 implementation, tests, data pipeline, training scripts

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Oct 30, 2026) Browse files

30 changed files +20,001 −0

30 changed files
Load all 30 files Unified Split
added .gitignore +24 −0
@@ -0,0 +1,24 @@
1 +.venv/
2 +__pycache__/
3 +*.pyc
4 +.pytest_cache/
5 +.hypothesis/
6 +experiments/runs/**/*.pt
7 +experiments/runs/**/*.safetensors
8 +experiments/runs/**/*.bin
9 +data/raw/
10 +data/cache/
11 +data/processed/*.bin
12 +data/processed/*.npy
13 +*.log.gz
14 +paper/*.aux
15 +paper/*.bbl
16 +paper/*.blg
17 +paper/*.fls
18 +paper/*.fdb_latexmk
19 +paper/*.out
20 +paper/*.toc
21 +paper/*.synctex.gz
22 +paper/*.lof
23 +paper/*.lot
24 +.DS_Store
added CLAUDE.md +15 −0
@@ -0,0 +1,15 @@
1 +# CLAUDE.md — WP31: PRISME-SLM (execution mandate)
2 +
3 +This repository executes UQO Working Paper 31, *PRISME: Intervention-Supervised Sparse Workspace Editing for
4 +Small Language Models* (author: Simon-Pierre Boucher, UQO). The full research mandate (phases 0–15, immutable
5 +rules, hypotheses H1–H7, baselines, ablations, paper structure) was supplied by the author on 2026-10-29 and is
6 +reproduced in `docs/MANDATE.md`. Working rules that bind every session in this repository:
7 +
8 +1. Scientific truth first: never fabricate runs, metrics, citations, hardware or proofs. Every number in the
9 + manuscript traces to a run id in `experiments/runs/` and `results/claim_registry.csv`.
10 +2. Phases are executed in order with acceptance gates logged in `PROGRESS.md`; failures go to `FAILURES.md`,
11 + design changes to `DECISIONS.md`. Unfavourable runs are never deleted.
12 +3. Minimal falsifiable version first (no persistent memory, no RAG, no MoE, no RL, no tool use).
13 +4. Resource boundaries: laptop (M5 Max) + existing MacLustr SSH access only; no purchases, no new credentials.
14 +5. Independent train / calibration / dev / locked-test splits; test splits are not read before
15 + `protocol/preregistration.md` is frozen (see `protocol/locked_test_manifest.json`).
added DECISIONS.md +13 −0
@@ -0,0 +1,13 @@
1 +# DECISIONS.md — design decisions (dated)
2 +
3 +| id | date | decision | rationale |
4 +|---|---|---|---|
5 +| D1 | 2026-10-29 | Repository lives at `Recherche-UQO/ia/prisme-slm` (new `ia/` category of the UQO series); Python 3.12 venv via `uv`, PyTorch 2.15 (CPU/MPS), fp32 training. | Series convention; bf16 autocast gave no speed-up on MPS (15.3k vs 15.0k tok/s on M3 Ultra). |
6 +| D2 | 2026-10-29 | A dedicated `bank_norm` RMSNorm precedes the bank/z projections. | The pre-final-norm residual stream has growing norm; projecting it raw would couple editor dynamics to residual scale. Costs d parameters. |
7 +| D3 | 2026-10-29 | Workspace/controller parameter counts are NOT forced to the brief's figures. | The brief's figures are reproduced exactly by a closed form (`4dw+Mw+17w²+6wf_w+15.5w+4`); the implemented design needs 7 fewer w×w projections per config (bank is read directly; readout uses raw slot values; controller shares one state MLP across the 24 actions so that scoring costs O(A·w²) not O(A·w·f_w)). Reconciliation reported in `results/tables/param_counts.json`. |
8 +| D4 | 2026-10-29 | STOP is absorbing in forced-action training too. | Needed for incremental/full-prefix equivalence (test revealed the mismatch). |
9 +| D5 | 2026-10-29 | Cost proxy c(a) in the controller's Q is the analytic ops ratio of the action (not a learned ĉ); the ĉ head is kept as a diagnostic regressor. | Per-action cost is deterministic given k(a) and T; learning it adds noise without information. |
10 +| D6 | 2026-10-29 | Synthetic splits are deduplicated by exact text across splits (train written first); eval items colliding with train are dropped and counted. | Low-depth cells have few distinct items; 995 train dups, 256 dev_iid dups removed. |
11 +| D7 | 2026-10-29 | Phase 4 uses a character-level tokenizer and a 2.1M-parameter backbone trained from scratch on the synthetic train split. | Self-contained learnability test independent of the web corpus/tokenizer. |
12 +| D8 | 2026-10-29 | 103M pilot backbone: 196.6M tokens (12k steps × 32 × 512) of FineWeb-Edu, one seed; parameter-matched plain decoder baseline (ffn 2144, 102.16M) trained with the identical recipe on a second node. | ~3.7 h per run at 14.8k tok/s on an M3 Ultra; seeds are varied at the editor/controller stage where the pilot's decisive comparison lives. |
13 +| D9 | 2026-10-29 | `readout_scale`: Δ = readout_scale · U(r(S_f) − r(S0)), with readout_scale a fixed constant set once at bootstrap to the measured rms of the backbone residual stream (5.6 for the tiny backbone). | Scale mismatch blocked the U = 0 bootstrap (see FAILURES). The final RMSNorm makes logits invariant to the joint scale of h and Δ, so this is a reparametrisation of U's learning rate; identity properties are unchanged (tests pass). |
added FAILURES.md +7 −0
@@ -0,0 +1,7 @@
1 +# FAILURES.md — failed hypotheses, tests and attempts (never deleted)
2 +
3 +| date | what failed | evidence | resolution |
4 +|---|---|---|---|
5 +| 2026-10-29 | `test_incremental_equals_full_prefix_forced`: cached decoding ≠ full-prefix forward. | pytest, atol 1e-4 | Root cause: forced-action forward let edits resume after STOP; made STOP absorbing (D4). |
6 +| 2026-10-29 | rsync exclusion `experiments` also dropped `configs/experiments/`; baseline launch on M3U96b failed twice (`FileNotFoundError`). | `experiments/logs/pretrain_enlarged102m_seed1.log` (first attempts) | anchored pattern `/experiments`. |
7 +| 2026-10-29 | Stage B bootstrap degeneracy (brief §13.1 confirmed): with U = 0 and lr 1e-3, 700 steps of forced mixed-budget training left val loss identical at every budget (0.4144 / 0.4143 / 0.4143 / 0.4146 / 0.4164 for budgets 0/1/2/4/8); grad-norm 0.01–0.02. Diagnosis: scale mismatch — backbone residual rms 5.64 vs readout difference rms 0.06, so U must grow by ×100 before edits can affect logits. | `experiments/runs/p4_editor_v1_seed01_*/metrics.jsonl` (stopped at step ~750) | D9: fixed non-learned `readout_scale` = measured backbone residual rms at bootstrap (keeps U = 0 init and exact identity; equivalent to a scale-aware learning rate). Variants with nonzero U init (std 0.02) and lr 3e-3 kept running on M2U64 / M4M64c as evidence. |
added Makefile +10 −0
@@ -0,0 +1,10 @@
1 +PY=.venv/bin/python
2 +.PHONY: env test audit params flops paper figures smoke
3 +env: ; uv venv --python 3.12 .venv && uv pip install -p .venv/bin/python -r requirements-lock.txt
4 +test: ; $(PY) -m pytest tests -q
5 +audit: ; $(PY) scripts/environment_audit.py --smoke
6 +params: ; $(PY) scripts/count_params.py --all-configs --device meta
7 +flops: ; $(PY) scripts/estimate_flops.py --config configs/prisme_459m.yaml --T 4096
8 +smoke: ; $(PY) -m training.pretrain --config configs/experiments/pretrain_103m.yaml --dry-run
9 +figures: ; $(PY) scripts/reproduce_figures.py --results results/raw/
10 +paper: ; cd paper && latexmk -pdf -interaction=nonstopmode main.tex
added PROGRESS.md +10 −0
@@ -0,0 +1,10 @@
1 +# PROGRESS.md — WP31 PRISME phase log
2 +
3 +| date | phase | status | commands / evidence | measurements | gate | blockers / next |
4 +|---|---|---|---|---|---|---|
5 +| 2026-10-29 | 0 env audit | done | `scripts/environment_audit.py --smoke` → `hardware/device_inventory.json`; `scripts/cluster_audit.sh` (read-only SSH) | laptop M5 Max 18c/40 GPU/48 GB, torch 2.15 MPS+CPU smoke OK; 12 LAN nodes reachable incl. 2× M3 Ultra 96 GB, M2 Ultra 64 GB, 3× M4 Max 64 GB | PASS | — |
6 +| 2026-10-29 | 1 literature | done | `literature/papers.bib` (77 verified entries), `comparison_matrix.csv`, `novelty_audit.md` | all 12 mandated refs exist incl. RecurTrace 2609.03379 and ELR 2610.12582; C3 (differential zero-init bypass) judged not novel; C2 novel only in its multi-action+propensity form | PASS | preregistration frozen before any test split is read (see protocol/) |
7 +| 2026-10-29 | 2 backbone + counts | done | `scripts/count_params.py --all-configs --device meta`; `scripts/estimate_flops.py` | backbone counts match brief exactly (99,502,848 / 446,744,064 / 1,149,339,648); workspace 2,631,173 vs brief 3,088,260 (reconciled, D3); ops table reproduces brief within 1.3 % | PASS | — |
8 +| 2026-10-29 | 3 invariants | done | `python -m pytest tests -q` → 22 passed | 15 mandated invariants + controller target-freedom + dense ablation + 5 cache-equivalence tests | PASS | — |
9 +| 2026-10-29 | 4 learnability | running | `training.train_editor --config configs/experiments/p4_tiny_backbone.yaml --device mps` (laptop) | — | — | then editor bootstrap on frozen tiny backbone |
10 +| 2026-10-29 | 8 pilot (Stage A) | running | M3U96a: `pretrain_103m.yaml` (run `pretrain103m_seed01_*`); M3U96b: `pretrain_enlarged_102m.yaml` | 14.8k tok/s, 196.6M tokens planned (~3.7 h each) | — | editor/controller stages after completion |
added baselines/__init__.py +0 −0
added baselines/global_recurrence.py +78 −0
@@ -0,0 +1,78 @@
1 +"""Global latent-recurrence adapter (Huginn/Ouro/ELR-*inspired*, simplified): the whole top representation is
2 +refined r times by ONE shared block operating at workspace width w, with a zero-initialised residual back to d.
3 +
4 +Same frozen backbone, same forced mixed-budget training recipe and same budget semantics as PRISME; the only
5 +difference is that every "action" rewrites the entire latent (global, dense) instead of 1-2 of 16 slots, and
6 +there is no slot selection to learn. Disclosed simplifications: no prelude/coda retraining, no truncated
7 +backprop through time, adapter-level recurrence on a frozen backbone rather than from-scratch recurrent pretraining.
8 +Parameter count is matched to the PRISME workspace (ffn width chosen in the experiment config)."""
9 +from __future__ import annotations
10 +
11 +import torch
12 +import torch.nn as nn
13 +
14 +from prisme.attention import RMSNorm, SwiGLU
15 +from prisme.backbone import Backbone
16 +from prisme.config import PrismeConfig
17 +import torch.nn.functional as F
18 +
19 +
20 +class RecurrentBlock(nn.Module):
21 + def __init__(self, w: int, heads: int, ffn: int, eps: float):
22 + super().__init__()
23 + self.h, self.hd = heads, w // heads
24 + self.n1 = RMSNorm(w, eps); self.q = nn.Linear(w, w, bias=False); self.k = nn.Linear(w, w, bias=False)
25 + self.v = nn.Linear(w, w, bias=False); self.o = nn.Linear(w, w, bias=False)
26 + self.n2 = RMSNorm(w, eps); self.ffn = SwiGLU(w, ffn)
27 +
28 + def forward(self, x, ctx_mask):
29 + B, T, w = x.shape
30 + xn = self.n1(x)
31 + q = self.q(xn).view(B, T, self.h, self.hd).transpose(1, 2)
32 + k = self.k(xn).view(B, T, self.h, self.hd).transpose(1, 2)
33 + v = self.v(xn).view(B, T, self.h, self.hd).transpose(1, 2)
34 + a = F.scaled_dot_product_attention(q, k, v, attn_mask=ctx_mask)
35 + x = x + self.o(a.transpose(1, 2).reshape(B, T, w))
36 + return x + self.ffn(self.n2(x))
37 +
38 +
39 +class GlobalRecurrenceModel(nn.Module):
40 + """Interface-compatible with PrismeModel.forward(tokens, actions): any nonterminal action = one global
41 + refinement step; STOP = no step. n_edits semantics identical."""
42 +
43 + def __init__(self, cfg: PrismeConfig, ffn_w: int | None = None, heads: int | None = None):
44 + super().__init__()
45 + self.cfg = cfg
46 + self.backbone = Backbone(cfg)
47 + w = cfg.w
48 + self.bank_norm = RMSNorm(cfg.d_model, cfg.norm_eps)
49 + self.down = nn.Linear(cfg.d_model, w, bias=False)
50 + self.block = RecurrentBlock(w, heads or cfg.n_editor_heads, ffn_w or cfg.ffn_w, cfg.norm_eps)
51 + self.ctx_proj = nn.Linear(2, w, bias=True)
52 + self.up = nn.Linear(w, cfg.d_model, bias=False)
53 + nn.init.zeros_(self.up.weight); nn.init.zeros_(self.ctx_proj.weight); nn.init.zeros_(self.ctx_proj.bias)
54 + self.controller = nn.Identity() # placeholder so parameter-group helpers work
55 +
56 + def forward(self, tokens, actions=None, return_state=False):
57 + cfg = self.cfg
58 + H = self.backbone(tokens)
59 + if actions is None or actions.shape[-1] == 0:
60 + return {"logits": self.backbone.head(H), "H": H, "n_edits": None}
61 + B, T, R = actions.shape
62 + x0 = self.down(self.bank_norm(H))
63 + x = x0
64 + mask = torch.ones(T, T, dtype=torch.bool, device=H.device).tril().view(1, 1, T, T)
65 + stopped = torch.zeros(B, T, dtype=torch.bool, device=H.device)
66 + n_edits = torch.zeros(B, T, dtype=torch.long, device=H.device)
67 + for r in range(R):
68 + a = torch.where(stopped, torch.full_like(actions[:, :, r], cfg.stop_action), actions[:, :, r])
69 + act = (a != cfg.stop_action)
70 + stopped |= ~act
71 + ctx = torch.stack([torch.full((B, T), r / cfg.r_max, device=H.device), torch.full((B, T), (R - r - 1) / cfg.r_max, device=H.device)], -1)
72 + xn = self.block(x + self.ctx_proj(ctx), mask)
73 + x = torch.where(act.unsqueeze(-1), xn, x) # positions that stopped keep their state
74 + n_edits += act.long()
75 + delta = self.up(x - x0)
76 + edited = n_edits > 0
77 + delta = torch.where(edited.unsqueeze(-1), delta, torch.zeros_like(delta))
78 + return {"logits": self.backbone.head(H + delta), "H": H, "n_edits": n_edits, "delta_out": delta}
added baselines/routers.py +54 −0
@@ -0,0 +1,54 @@
1 +"""Reference routing policies that share PRISME's editor (actions chosen without the learned value controller).
2 +All are target-free and deployable; they differ only in how the action is picked at each step.
3 + random uniform over the 24 nonterminal actions, fixed budget b
4 + uniform round-robin single-slot schedule SINGLE_0, SINGLE_1, ... (deterministic, fixed budget b)
5 + fixed_pair PAIR_0_1 repeated (fixed budget b)
6 + entropy continue (random action) while the current predictive entropy exceeds a threshold, else STOP
7 + norm continue while the last write moved the slot by more than a threshold (RMS), else STOP
8 +"""
9 +from __future__ import annotations
10 +
11 +import torch
12 +
13 +from prisme.model import PrismeModel
14 +
15 +
16 +@torch.no_grad()
17 +def rollout(model: PrismeModel, tokens, policy: str, budget: int, threshold: float = 0.0, seed: int = 0):
18 + cfg = model.cfg
19 + ctx = model.encode(tokens)
20 + S = ctx.S0
21 + B, T = tokens.shape
22 + g = torch.Generator(device="cpu").manual_seed(seed)
23 + stopped = torch.zeros(B, T, dtype=torch.bool, device=tokens.device)
24 + n_edits = torch.zeros(B, T, dtype=torch.long, device=tokens.device)
25 + trace = torch.full((B, T, budget), cfg.stop_action, dtype=torch.long, device=tokens.device)
26 + last_rms = torch.full((B, T), float("inf"), device=tokens.device)
27 + for r in range(budget):
28 + if policy == "random":
29 + a = torch.randint(0, cfg.n_actions, (B, T), generator=g).to(tokens.device)
30 + elif policy == "uniform":
31 + a = torch.full((B, T), r % cfg.n_slots, device=tokens.device)
32 + elif policy == "fixed_pair":
33 + a = torch.full((B, T), cfg.n_slots + (r % cfg.n_pairs), device=tokens.device)
34 + elif policy == "entropy":
35 + logits, _ = model.logits_from_state(S, ctx, n_edits > 0)
36 + p = torch.softmax(logits.float(), -1)
37 + ent = -(p * torch.log(p.clamp_min(1e-12))).sum(-1)
38 + a = torch.randint(0, cfg.n_actions, (B, T), generator=g).to(tokens.device)
39 + a = torch.where(ent > threshold, a, torch.full_like(a, cfg.stop_action))
40 + elif policy == "norm":
41 + a = torch.randint(0, cfg.n_actions, (B, T), generator=g).to(tokens.device)
42 + a = torch.where(last_rms > threshold, a, torch.full_like(a, cfg.stop_action))
43 + else:
44 + raise ValueError(policy)
45 + a = torch.where(stopped, torch.full_like(a, cfg.stop_action), a)
46 + stopped |= a == cfg.stop_action
47 + trace[:, :, r] = a
48 + S_new, info = model.apply_action(S, ctx, a, r, budget - r - 1)
49 + moved = (info["eta"].squeeze(-1) * info["delta_rms"] * info["scale"]) # [B,T,k] realised rms step per lane
50 + last_rms = torch.where(a != cfg.stop_action, moved.max(-1).values, last_rms)
51 + S = S_new
52 + n_edits += (a != cfg.stop_action).long()
53 + logits, _ = model.logits_from_state(S, ctx, n_edits > 0)
54 + return {"logits": logits, "actions": trace, "n_edits": n_edits}
added configs/baselines/transformer_enlarged_102m.yaml +23 −0
@@ -0,0 +1,23 @@
1 +# PRISME reference configuration: prisme_103m (see docs/architecture_spec.md)
2 +model:
3 + name: transformer_enlarged_102m
4 + vocab_size: 32768
5 + d_model: 768
6 + n_layers: 12
7 + n_heads: 12
8 + n_kv_heads: 3
9 + ffn_width: 2144 # parameter-matched plain decoder: +2.65M params ~= PRISME workspace+controller (2.63M)
10 + max_seq_len: 4096
11 + rope_theta: 10000.0
12 + norm_eps: 1.0e-5
13 + tie_embeddings: true
14 + use_workspace: false
15 + w: 256
16 + ffn_w: 768
17 + n_slots: 16
18 + n_editor_blocks: 2
19 + n_editor_heads: 4
20 + r_max: 8
21 + eta_max: 0.25
22 + clip_c: 1.0
23 + readout_zero_init: true
added configs/experiments/controller_p4.yaml +14 −0
@@ -0,0 +1,14 @@
1 +# Phase 6 (tiny scale): value controller fit on logged labels; (lambda, kappa) tuned on calibration rollouts.
2 +name: controller_p4
3 +seed: 1
4 +mode: prisme
5 +model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512,
6 + max_seq_len: 320, w: 64, ffn_w: 192, n_slots: 16, n_editor_heads: 2, r_max: 8}
7 +checkpoint: CHECKPOINT_DIR
8 +labels: {train: LABELS_TRAIN, calib: LABELS_CALIB}
9 +train: {steps: 3000, lr: 1.0e-3, weight_decay: 0.01, alpha: 1.0, beta: 0.5, gamma: 0.1, huber_delta: 0.1, log_every: 100, eval_every: 1000, seq_len: 320}
10 +lambda_grid: [0.0, 0.01, 0.03, 0.1, 0.3]
11 +kappa_grid: [0.0, 0.5]
12 +sweep_batches: 10
13 +max_budget: 8
14 +selection_rule: max_loss_reduction_per_ops
added configs/experiments/labels_p4.yaml +13 −0
@@ -0,0 +1,13 @@
1 +# Phase 5 (tiny scale): counterfactual label collection with the fixed Phase 4 editor checkpoint.
2 +name: labels_p4
3 +seed: 1
4 +mode: prisme
5 +model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512,
6 + max_seq_len: 320, w: 64, ffn_w: 192, n_slots: 16, n_editor_heads: 2, r_max: 8}
7 +checkpoint: CHECKPOINT_DIR
8 +use_policy: false
9 +data: {kind: synthetic, tokenizer: char, splits_dir: data/splits/synthetic_v1, loss_span: answer}
10 +collect: {batch_size: 32, seq_len: 320, max_prefix: 3, audit_every: 4}
11 +partitions:
12 + train: {split: train, n_batches: 120, stream_seed: 11}
13 + calib: {split: calib, n_batches: 30, stream_seed: 12}
added configs/experiments/p4_editor_bootstrap.yaml +16 −0
@@ -0,0 +1,16 @@
1 +# Phase 4 step 2: Stage B editor bootstrap on the FROZEN tiny backbone (v1), forced mixed budgets {0,1,2,4,8}.
2 +name: p4_editor_v1_scaled
3 +seed: 1
4 +mode: prisme
5 +model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512,
6 + max_seq_len: 320, w: 64, ffn_w: 192, n_slots: 16, n_editor_heads: 2, r_max: 8}
7 +init: {backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint, freeze_backbone: true, auto_readout_scale: true}
8 +data: {kind: synthetic, tokenizer: char, splits_dir: data/splits/synthetic_v1, loss_span: answer,
9 + eval_splits: [dev_iid, dev_template, dev_depth, dev_compose]}
10 +budgets: [0, 1, 2, 4, 8]
11 +budget_probs: [0.2, 0.2, 0.2, 0.2, 0.2]
12 +budget_per: position
13 +train: {batch_size: 32, seq_len: 320, steps: 3000, lr: 1.0e-3, warmup: 100, weight_decay: 0.1, grad_clip: 1.0,
14 + log_every: 50, eval_every: 500, eval_batches: 8, ckpt_every: 1500}
15 +em_budgets: [0, 2, 8]
16 +em_max_items: 400
added configs/experiments/p4_editor_bootstrap_lr3e3.yaml +16 −0
@@ -0,0 +1,16 @@
1 +# Phase 4 step 2: Stage B editor bootstrap on the FROZEN tiny backbone (v1), forced mixed budgets {0,1,2,4,8}.
2 +name: p4_editor_v1_lr3e3_b64
3 +seed: 1
4 +mode: prisme
5 +model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512,
6 + max_seq_len: 320, w: 64, ffn_w: 192, n_slots: 16, n_editor_heads: 2, r_max: 8}
7 +init: {backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint, freeze_backbone: true}
8 +data: {kind: synthetic, tokenizer: char, splits_dir: data/splits/synthetic_v1, loss_span: answer,
9 + eval_splits: [dev_iid, dev_template, dev_depth, dev_compose]}
10 +budgets: [0, 1, 2, 4, 8]
11 +budget_probs: [0.2, 0.2, 0.2, 0.2, 0.2]
12 +budget_per: position
13 +train: {batch_size: 64, seq_len: 320, steps: 2000, lr: 3.0e-3, warmup: 100, weight_decay: 0.1, grad_clip: 1.0,
14 + log_every: 50, eval_every: 500, eval_batches: 8, ckpt_every: 1500}
15 +em_budgets: [0, 2, 8]
16 +em_max_items: 400
added configs/experiments/p4_editor_bootstrap_nonzeroU.yaml +17 −0
@@ -0,0 +1,17 @@
1 +# Phase 4 step 2: Stage B editor bootstrap on the FROZEN tiny backbone (v1), forced mixed budgets {0,1,2,4,8}.
2 +name: p4_editor_v1_nonzeroU
3 +seed: 1
4 +mode: prisme
5 +model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512,
6 + max_seq_len: 320, w: 64, ffn_w: 192, n_slots: 16, n_editor_heads: 2, r_max: 8}
7 +init: {backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint, freeze_backbone: true}
8 +data: {kind: synthetic, tokenizer: char, splits_dir: data/splits/synthetic_v1, loss_span: answer,
9 + eval_splits: [dev_iid, dev_template, dev_depth, dev_compose]}
10 +budgets: [0, 1, 2, 4, 8]
11 +budget_probs: [0.2, 0.2, 0.2, 0.2, 0.2]
12 +budget_per: position
13 +train: {batch_size: 32, seq_len: 320, steps: 2000, lr: 1.0e-3, warmup: 100, weight_decay: 0.1, grad_clip: 1.0,
14 + log_every: 50, eval_every: 500, eval_batches: 8, ckpt_every: 1500}
15 +em_budgets: [0, 2, 8]
16 +em_max_items: 400
17 +model_overrides: {readout_zero_init: false}
added configs/experiments/p4_tiny_backbone.yaml +12 −0
@@ -0,0 +1,12 @@
1 +# Phase 4 step 1: tiny backbone pretrained from scratch on the synthetic TRAIN split (char-level), workspace off.
2 +name: p4_tiny_backbone
3 +seed: 1
4 +mode: backbone_only
5 +model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512,
6 + max_seq_len: 320, w: 64, ffn_w: 192, n_slots: 16, n_editor_heads: 2, r_max: 8}
7 +init: {freeze_backbone: false}
8 +data: {kind: synthetic, tokenizer: char, splits_dir: data/splits/synthetic_v1, loss_span: all,
9 + eval_splits: [dev_iid, dev_template, dev_depth, dev_compose]}
10 +train: {batch_size: 64, seq_len: 320, steps: 6000, lr: 1.0e-3, warmup: 200, weight_decay: 0.1, grad_clip: 1.0,
11 + log_every: 50, eval_every: 500, eval_batches: 10, ckpt_every: 2000}
12 +em_max_items: 600
added configs/experiments/p4_tiny_backbone_v2.yaml +12 −0
@@ -0,0 +1,12 @@
1 +# Phase 4 step 1 (v2): stronger tiny backbone (≈5.6M params) from scratch on the synthetic TRAIN split, char-level.
2 +name: p4_tiny_backbone_v2
3 +seed: 1
4 +mode: backbone_only
5 +model: {name: tiny_p4v2, vocab_size: 128, d_model: 256, n_layers: 6, n_heads: 8, n_kv_heads: 4, ffn_width: 768,
6 + max_seq_len: 320, w: 64, ffn_w: 192, n_slots: 16, n_editor_heads: 2, r_max: 8}
7 +init: {freeze_backbone: false}
8 +data: {kind: synthetic, tokenizer: char, splits_dir: data/splits/synthetic_v1, loss_span: all,
9 + eval_splits: [dev_iid, dev_template, dev_depth, dev_compose]}
10 +train: {batch_size: 64, seq_len: 320, steps: 15000, lr: 1.0e-3, warmup: 300, weight_decay: 0.1, grad_clip: 1.0,
11 + log_every: 100, eval_every: 1000, eval_batches: 10, ckpt_every: 5000}
12 +em_max_items: 800
added configs/experiments/pretrain_103m.yaml +18 −0
@@ -0,0 +1,18 @@
1 +name: pretrain103m
2 +seed: 1
3 +model: configs/prisme_103m.yaml
4 +data:
5 + train_bin: data/processed/fineweb_edu_300m/train.bin
6 + val_bin: data/processed/fineweb_edu_300m/val.bin
7 +train:
8 + batch_size: 32
9 + seq_len: 512
10 + steps: 12000 # 32*512*12000 = 196.6M tokens (pilot budget; see protocol/preregistration.md)
11 + lr: 6.0e-4
12 + warmup: 300
13 + weight_decay: 0.1
14 + grad_clip: 1.0
15 + log_every: 20
16 + eval_every: 500
17 + eval_batches: 20
18 + ckpt_every: 1000
added configs/experiments/pretrain_enlarged_102m.yaml +18 −0
@@ -0,0 +1,18 @@
1 +name: pretrain_enlarged102m
2 +seed: 1
3 +model: configs/baselines/transformer_enlarged_102m.yaml
4 +data:
5 + train_bin: data/processed/fineweb_edu_300m/train.bin
6 + val_bin: data/processed/fineweb_edu_300m/val.bin
7 +train:
8 + batch_size: 32
9 + seq_len: 512
10 + steps: 12000 # 32*512*12000 = 196.6M tokens (pilot budget; see protocol/preregistration.md)
11 + lr: 6.0e-4
12 + warmup: 300
13 + weight_decay: 0.1
14 + grad_clip: 1.0
15 + log_every: 20
16 + eval_every: 500
17 + eval_batches: 20
18 + ckpt_every: 1000
added configs/experiments/routing_p4.yaml +13 −0
@@ -0,0 +1,13 @@
1 +# Routing comparison (tiny scale) on dev splits with the LOCKED controller checkpoint.
2 +seed: 1
3 +mode: prisme
4 +model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512,
5 + max_seq_len: 320, w: 64, ffn_w: 192, n_slots: 16, n_editor_heads: 2, r_max: 8}
6 +checkpoint: CHECKPOINT_DIR
7 +data: {kind: synthetic, tokenizer: char, splits_dir: data/splits/synthetic_v1, loss_span: answer, split: dev_iid}
8 +batch_size: 32
9 +seq_len: 320
10 +n_batches: 20
11 +budgets: [1, 2, 4, 8]
12 +oracle_batches: 4
13 +two_step_batches: 1
added configs/prisme_103m.yaml +23 −0
@@ -0,0 +1,23 @@
1 +# PRISME reference configuration: prisme_103m (see docs/architecture_spec.md)
2 +model:
3 + name: prisme_103m
4 + vocab_size: 32768
5 + d_model: 768
6 + n_layers: 12
7 + n_heads: 12
8 + n_kv_heads: 3
9 + ffn_width: 2048
10 + max_seq_len: 4096
11 + rope_theta: 10000.0
12 + norm_eps: 1.0e-5
13 + tie_embeddings: true
14 + use_workspace: true
15 + w: 256
16 + ffn_w: 768
17 + n_slots: 16
18 + n_editor_blocks: 2
19 + n_editor_heads: 4
20 + r_max: 8
21 + eta_max: 0.25
22 + clip_c: 1.0
23 + readout_zero_init: true
added configs/prisme_1180m.yaml +23 −0
@@ -0,0 +1,23 @@
1 +# PRISME reference configuration: prisme_1180m (see docs/architecture_spec.md)
2 +model:
3 + name: prisme_1180m
4 + vocab_size: 32768
5 + d_model: 2048
6 + n_layers: 24
7 + n_heads: 32
8 + n_kv_heads: 8
9 + ffn_width: 5632
10 + max_seq_len: 4096
11 + rope_theta: 10000.0
12 + norm_eps: 1.0e-5
13 + tie_embeddings: true
14 + use_workspace: true
15 + w: 768
16 + ffn_w: 2048
17 + n_slots: 16
18 + n_editor_blocks: 2
19 + n_editor_heads: 12
20 + r_max: 8
21 + eta_max: 0.25
22 + clip_c: 1.0
23 + readout_zero_init: true
added configs/prisme_459m.yaml +23 −0
@@ -0,0 +1,23 @@
1 +# PRISME reference configuration: prisme_459m (see docs/architecture_spec.md)
2 +model:
3 + name: prisme_459m
4 + vocab_size: 32768
5 + d_model: 1536
6 + n_layers: 16
7 + n_heads: 24
8 + n_kv_heads: 6
9 + ffn_width: 4096
10 + max_seq_len: 4096
11 + rope_theta: 10000.0
12 + norm_eps: 1.0e-5
13 + tie_embeddings: true
14 + use_workspace: true
15 + w: 512
16 + ffn_w: 1536
17 + n_slots: 16
18 + n_editor_blocks: 2
19 + n_editor_heads: 8
20 + r_max: 8
21 + eta_max: 0.25
22 + clip_c: 1.0
23 + readout_zero_init: true
added data/__init__.py +0 −0
added data/char_tokenizer.py +29 −0
@@ -0,0 +1,29 @@
1 +"""Self-contained character-level tokenizer for the Phase 4 learnability experiments (no corpus dependency)."""
2 +from __future__ import annotations
3 +
4 +import json
5 +from pathlib import Path
6 +
7 +PRINTABLE = [chr(i) for i in range(32, 127)]
8 +
9 +
10 +class CharTokenizer:
11 + def __init__(self):
12 + self.specials = ["<eot>", "<pad>"]
13 + self.itos = self.specials + PRINTABLE
14 + self.stoi = {s: i for i, s in enumerate(self.itos)}
15 + self.eot_id = 0
16 + self.pad_id = 1
17 +
18 + @property
19 + def vocab_size(self) -> int:
20 + return len(self.itos)
21 +
22 + def encode(self, text: str) -> list[int]:
23 + return [self.stoi.get(ch, self.stoi["?"]) for ch in text]
24 +
25 + def decode(self, ids) -> str:
26 + return "".join(self.itos[i] if i >= len(self.specials) else "" for i in ids)
27 +
28 + def save(self, path):
29 + Path(path).write_text(json.dumps({"itos": self.itos}))
added data/generators/__init__.py +0 −0
added data/generators/synthetic.py +236 −0
@@ -0,0 +1,236 @@
1 +"""Verifiable synthetic task generators with programmatic ground truth.
2 +
3 +Families (each item is a dict: family, template, depth, prompt, answer, text, meta):
4 + arith multi-step arithmetic chains (depth = number of operations)
5 + bind variable binding + dependent substitution (depth = chain length)
6 + fsm register / finite-state updates (depth = number of updates)
7 + excl conjunctive constraints with distractors (depth = number of constraints)
8 + sched rule-based selection under a budget and exclusions (depth = number of candidates)
9 + compose held-out composition: bind chain feeding an arithmetic expression (depth = total steps)
10 +
11 +Splitting is done BEFORE sampling: by (family, template id, depth regime, seed stream). Train uses templates
12 +T0-T2 and depths <= max_train_depth; held-out "template" items use templates T3-T4; held-out "depth" items use
13 +depths > max_train_depth; the compose family is entirely held out from training unless explicitly requested.
14 +Text format: "<prompt> Answer: <answer>" ; the answer is the only supervised span for exact-match evaluation.
15 +"""
16 +from __future__ import annotations
17 +
18 +import hashlib
19 +import json
20 +import random
21 +from dataclasses import dataclass, asdict
22 +from typing import Callable
23 +
24 +OPS = {"+": lambda a, b: a + b, "-": lambda a, b: a - b, "*": lambda a, b: a * b}
25 +
26 +# Phrasing templates: index 0..2 = train, 3..4 = held-out paraphrases.
27 +TEMPLATES = {
28 + "arith": ["Compute {expr}.", "What is {expr}?", "Evaluate the expression {expr}.", "Calculate {expr} step by step and give the result.", "Find the value of {expr}."],
29 + "bind": ["Let {chain}. What is {var}?", "Given {chain}, compute {var}.", "{chain}. Find {var}.", "Suppose {chain}. Then {var} equals what?", "With {chain}, the value of {var} is?"],
30 + "fsm": ["Start with register {init}. {ops}. What is the register?", "The counter begins at {init}. {ops}. Final counter?", "Initial value {init}. {ops}. Result?", "A register holds {init}. Apply: {ops}. What does it hold now?", "Begin at {init}; {ops}; final value?"],
31 + "excl": ["Items: {items}. Keep the items that are {rules}. Which items remain?", "Consider {items}. Select those that are {rules}. List them.", "{items}. Filter: {rules}. Remaining items?", "Given the list {items}, keep only the ones {rules}. Answer with the names.", "From {items}, which satisfy: {rules}?"],
32 + "sched": ["Budget {budget}. Tasks: {tasks}. Rule: {rule}. Choose the affordable task with the highest value.", "You have {budget} units. Options: {tasks}. Constraint: {rule}. Which affordable option has the largest value?", "Capacity {budget}. Candidates: {tasks}. Exclusion: {rule}. Pick the best affordable candidate.", "With a budget of {budget}, among {tasks}, under the rule {rule}, which task maximises value while staying affordable?", "Budget is {budget}. {tasks}. {rule}. Best affordable task by value?"],
33 + "compose": ["Let {chain}. Compute {expr}.", "Given {chain}, what is {expr}?", "{chain}. Evaluate {expr}.", "Suppose {chain}. Then find {expr}.", "With {chain}, calculate {expr}."],
34 +}
35 +
36 +
37 +def _rng(family: str, template: int, depth: int, seed: int, idx: int) -> random.Random:
38 + key = f"{family}|{template}|{depth}|{seed}|{idx}"
39 + return random.Random(int(hashlib.sha256(key.encode()).hexdigest()[:16], 16))
40 +
41 +
42 +def gen_arith(r: random.Random, depth: int, template: int):
43 + vals = [r.randint(1, 20)]
44 + ops = []
45 + expr = str(vals[0])
46 + cur = vals[0]
47 + for _ in range(depth):
48 + op = r.choice(["+", "-", "*"])
49 + b = r.randint(1, 9) if op == "*" else r.randint(1, 30)
50 + cur = OPS[op](cur, b) # left-to-right evaluation (stated below)
51 + expr += f" {op} {b}"
52 + ops.append(op)
53 + prompt = TEMPLATES["arith"][template].format(expr=expr) + " Evaluate left to right."
54 + return prompt, str(cur), {"ops": "".join(ops)}
55 +
56 +
57 +def gen_bind(r: random.Random, depth: int, template: int):
58 + names = r.sample(list("abcdefghjkmnpqrstuvwxyz"), depth + 1)
59 + vals = {names[0]: r.randint(1, 15)}
60 + parts = [f"{names[0]} = {vals[names[0]]}"]
61 + for i in range(1, depth + 1):
62 + op = r.choice(["+", "-", "*"])
63 + b = r.randint(1, 5) if op == "*" else r.randint(1, 12)
64 + src = names[i - 1]
65 + vals[names[i]] = OPS[op](vals[src], b)
66 + parts.append(f"{names[i]} = {src} {op} {b}")
67 + # distractor: an unused variable
68 + d = r.choice([n for n in "ABCDEFG"])
69 + parts.insert(r.randint(1, len(parts)), f"{d} = {r.randint(1, 40)}")
70 + chain = ", ".join(parts)
71 + target = names[depth]
72 + prompt = TEMPLATES["bind"][template].format(chain=chain, var=target)
73 + return prompt, str(vals[target]), {"target": target}
74 +
75 +
76 +def gen_fsm(r: random.Random, depth: int, template: int):
77 + init = r.randint(0, 10)
78 + cur = init
79 + steps = []
80 + for _ in range(depth):
81 + op = r.choice(["add", "sub", "double", "reset", "halve"])
82 + if op == "add":
83 + b = r.randint(1, 9); cur += b; steps.append(f"add {b}")
84 + elif op == "sub":
85 + b = r.randint(1, 9); cur -= b; steps.append(f"subtract {b}")
86 + elif op == "double":
87 + cur *= 2; steps.append("double it")
88 + elif op == "halve":
89 + cur = cur // 2; steps.append("halve it (round down)")
90 + else:
91 + b = r.randint(0, 9); cur = b; steps.append(f"reset to {b}")
92 + prompt = TEMPLATES["fsm"][template].format(init=init, ops=", then ".join(steps))
93 + return prompt, str(cur), {}
94 +
95 +
96 +def gen_excl(r: random.Random, depth: int, template: int):
97 + n = r.randint(4, 7)
98 + names = r.sample(["apple", "pear", "plum", "fig", "kiwi", "lime", "date", "yam", "nut", "bean"], n)
99 + items = {nm: (r.randint(1, 30), r.choice(["red", "green", "blue"])) for nm in names}
100 + rules = []
101 + def keep(v, c):
102 + return True
103 + preds: list[Callable] = []
104 + for _ in range(depth):
105 + kind = r.choice(["gt", "lt", "color", "even"])
106 + if kind == "gt":
107 + t = r.randint(5, 25); rules.append(f"greater than {t}"); preds.append(lambda v, c, t=t: v > t)
108 + elif kind == "lt":
109 + t = r.randint(5, 25); rules.append(f"smaller than {t}"); preds.append(lambda v, c, t=t: v < t)
110 + elif kind == "color":
111 + col = r.choice(["red", "green", "blue"]); rules.append(f"not {col}"); preds.append(lambda v, c, col=col: c != col)
112 + else:
113 + rules.append("even"); preds.append(lambda v, c: v % 2 == 0)
114 + remaining = [nm for nm in names if all(p(*items[nm]) for p in preds)]
115 + items_s = ", ".join(f"{nm} ({v}, {c})" for nm, (v, c) in items.items())
116 + prompt = TEMPLATES["excl"][template].format(items=items_s, rules=" and ".join(rules))
117 + ans = " ".join(remaining) if remaining else "none"
118 + return prompt, ans, {"n_items": n, "n_remaining": len(remaining)}
119 +
120 +
121 +def gen_sched(r: random.Random, depth: int, template: int):
122 + names = [chr(ord("A") + i) for i in range(depth)]
123 + tasks = {nm: (r.randint(1, 12), r.randint(1, 20)) for nm in names}
124 + budget = r.randint(4, 12)
125 + kind = r.choice(["cost_gt", "value_lt", "none"])
126 + if kind == "cost_gt":
127 + t = r.randint(3, 10); rule = f"exclude tasks with cost greater than {t}"; ok = lambda c, v: c <= t
128 + elif kind == "value_lt":
129 + t = r.randint(3, 10); rule = f"exclude tasks with value below {t}"; ok = lambda c, v: v >= t
130 + else:
131 + rule = "no exclusion"; ok = lambda c, v: True
132 + feas = [(v, nm) for nm, (c, v) in tasks.items() if c <= budget and ok(c, v)]
133 + if not feas:
134 + ans = "none"
135 + else:
136 + best = max(v for v, _ in feas)
137 + ans = sorted(nm for v, nm in feas if v == best)[0]
138 + tasks_s = ", ".join(f"{nm} (cost {c}, value {v})" for nm, (c, v) in tasks.items())
139 + prompt = TEMPLATES["sched"][template].format(budget=budget, tasks=tasks_s, rule=rule)
140 + return prompt, ans, {"n_feasible": len(feas)}
141 +
142 +
143 +def gen_compose(r: random.Random, depth: int, template: int):
144 + # bind chain of length depth-2 feeding a 2-op arithmetic expression over two chain variables
145 + k = max(1, depth - 2)
146 + names = r.sample(list("abcdefghjkmnpqrstuvwxyz"), k + 1)
147 + vals = {names[0]: r.randint(1, 10)}
148 + parts = [f"{names[0]} = {vals[names[0]]}"]
149 + for i in range(1, k + 1):
150 + op = r.choice(["+", "-"]); b = r.randint(1, 9); src = names[i - 1]
151 + vals[names[i]] = OPS[op](vals[src], b); parts.append(f"{names[i]} = {src} {op} {b}")
152 + v1, v2 = r.sample(names, 2) if len(names) > 1 else (names[0], names[0])
153 + op1, op2 = r.choice(["+", "-", "*"]), r.choice(["+", "-"])
154 + c = r.randint(1, 9)
155 + val = OPS[op2](OPS[op1](vals[v1], vals[v2]), c)
156 + expr = f"{v1} {op1} {v2} {op2} {c} (left to right)"
157 + prompt = TEMPLATES["compose"][template].format(chain=", ".join(parts), expr=expr)
158 + return prompt, str(val), {}
159 +
160 +
161 +GENERATORS = {"arith": gen_arith, "bind": gen_bind, "fsm": gen_fsm, "excl": gen_excl, "sched": gen_sched, "compose": gen_compose}
162 +
163 +
164 +def make_item(family: str, template: int, depth: int, seed: int, idx: int) -> dict:
165 + r = _rng(family, template, depth, seed, idx)
166 + prompt, answer, meta = GENERATORS[family](r, depth, template)
167 + text = f"{prompt} Answer: {answer}"
168 + return {"family": family, "template": template, "depth": depth, "seed": seed, "idx": idx,
169 + "prompt": prompt + " Answer:", "answer": " " + answer, "text": text, "meta": meta,
170 + "uid": hashlib.sha256(text.encode()).hexdigest()[:16]}
171 +
172 +
173 +@dataclass
174 +class SplitSpec:
175 + name: str
176 + families: tuple[str, ...]
177 + templates: tuple[int, ...]
178 + depths: tuple[int, ...]
179 + seed: int
180 + n_per_cell: int
181 +
182 +
183 +def generate_split(spec: SplitSpec) -> list[dict]:
184 + items = []
185 + for fam in spec.families:
186 + for tpl in spec.templates:
187 + for dep in spec.depths:
188 + for i in range(spec.n_per_cell):
189 + items.append(make_item(fam, tpl, dep, spec.seed, i))
190 + return items
191 +
192 +
193 +def default_splits(n_train_per_cell: int = 400, n_eval_per_cell: int = 60) -> dict[str, SplitSpec]:
194 + """Train: 5 families, templates 0-2, depths 1-4, seed 1. Dev: seed 2 same cells (IID generalisation).
195 + Held-out template: templates 3-4. Held-out depth: depths 5-6. Compose: fully held-out family.
196 + Final test: seed 9 across all eval regimes (LOCKED until preregistration is frozen)."""
197 + fams = ("arith", "bind", "fsm", "excl", "sched")
198 + return {
199 + "train": SplitSpec("train", fams, (0, 1, 2), (1, 2, 3, 4), 1, n_train_per_cell),
200 + "calib": SplitSpec("calib", fams, (0, 1, 2), (1, 2, 3, 4), 3, n_eval_per_cell // 2),
201 + "dev_iid": SplitSpec("dev_iid", fams, (0, 1, 2), (1, 2, 3, 4), 2, n_eval_per_cell),
202 + "dev_template": SplitSpec("dev_template", fams, (3, 4), (1, 2, 3, 4), 2, n_eval_per_cell),
203 + "dev_depth": SplitSpec("dev_depth", fams, (0, 1, 2), (5, 6), 2, n_eval_per_cell),
204 + "dev_compose": SplitSpec("dev_compose", ("compose",), (0, 1, 2), (3, 4, 5), 2, n_eval_per_cell),
205 + "test_iid": SplitSpec("test_iid", fams, (0, 1, 2), (1, 2, 3, 4), 9, n_eval_per_cell),
206 + "test_template": SplitSpec("test_template", fams, (3, 4), (1, 2, 3, 4), 9, n_eval_per_cell),
207 + "test_depth": SplitSpec("test_depth", fams, (0, 1, 2), (5, 6), 9, n_eval_per_cell),
208 + "test_compose": SplitSpec("test_compose", ("compose",), (0, 1, 2), (3, 4, 5), 9, n_eval_per_cell),
209 + }
210 +
211 +
212 +def write_splits(out_dir: str, n_train_per_cell: int = 400, n_eval_per_cell: int = 60) -> dict:
213 + from pathlib import Path
214 + out = Path(out_dir); out.mkdir(parents=True, exist_ok=True)
215 + manifest = {}
216 + all_uids: dict[str, str] = {}
217 + for name, spec in default_splits(n_train_per_cell, n_eval_per_cell).items():
218 + items = generate_split(spec)
219 + # cross-split exact-duplicate removal: an item whose full text already appeared in an earlier split
220 + # (train is written first) is DROPPED from the later split; within-split duplicates are dropped too.
221 + kept, dups = [], 0
222 + for it in items:
223 + if it["uid"] in all_uids:
224 + dups += 1
225 + continue
226 + all_uids[it["uid"]] = name
227 + kept.append(it)
228 + items = kept
229 + p = out / f"{name}.jsonl"
230 + with open(p, "w") as f:
231 + for it in items:
232 + f.write(json.dumps(it) + "\n")
233 + manifest[name] = {"spec": asdict(spec), "n": len(items), "duplicates_dropped": dups,
234 + "sha256": hashlib.sha256(p.read_bytes()).hexdigest(), "locked": name.startswith("test")}
235 + (out / "manifest.json").write_text(json.dumps(manifest, indent=2))
236 + return manifest
added data/processed/fineweb_edu_300m/manifest.json +48 −0
@@ -0,0 +1,48 @@
1 +{
2 + "source": "HuggingFaceFW/fineweb-edu sample-10BT (streaming, canonical shard order)",
3 + "license": "ODC-By 1.0 (https://opendatacommons.org/licenses/by/1-0/)",
4 + "prepared_at": "2026-10-29T23:19:53-0400",
5 + "args": {
6 + "out": "data/processed/fineweb_edu_300m",
7 + "target_train_tokens": 300000000,
8 + "vocab_size": 32768,
9 + "tokenizer_docs": 60000,
10 + "val_pct": 0.01,
11 + "test_pct": 0.01,
12 + "max_doc_chars": 40000
13 + },
14 + "docs_streamed": 291647,
15 + "docs": {
16 + "train": 285604,
17 + "val": 2957,
18 + "test": 2973
19 + },
20 + "chars": {
21 + "train": 1290001347,
22 + "val": 12988848,
23 + "test": 12933032
24 + },
25 + "splits": {
26 + "train": {
27 + "tokens": 289959006,
28 + "docs": 285604,
29 + "sha256": "e931e83c4ba6e62a2125e50a5390e8df78913dbed31af805d8af17fafad95123",
30 + "locked": false
31 + },
32 + "val": {
33 + "tokens": 2913824,
34 + "docs": 2957,
35 + "sha256": "0565e3f4a4028b241f40ef07f20bc68b24e45d99e147860be066e3667d1532be",
36 + "locked": false
37 + },
38 + "test": {
39 + "tokens": 2883436,
40 + "docs": 2973,
41 + "sha256": "1ce9ca3ab6d659fe08042844fc0589951e09386ee79d2ef76504ffaaebb0947b",
42 + "locked": true
43 + }
44 + },
45 + "tokenizer_sha256": "5a78df618767945743623d5ff238d035839d02388b4c53c548a437b3d8a206ee",
46 + "eot_id": 0,
47 + "dtype": "uint16"
48 +}
\ No newline at end of file
added data/processed/fineweb_edu_300m/test.bin +0 −0

Binary file not shown.

added data/processed/fineweb_edu_300m/tokenizer.json +19236 −0
Large diff (19,236 lines) is not loaded by default to keep the page fast.Load diff

Diff truncated — file too large.

Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere