SPB Git

spb/wp31_uqo

Public
762 B · 13 lines yaml
1# Phase 4 step 1: tiny backbone pretrained from scratch on the synthetic TRAIN split (char-level), workspace off.2name: p4_tiny_backbone3seed: 14mode: backbone_only5model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512,6        max_seq_len: 320, w: 64, ffn_w: 192, n_slots: 16, n_editor_heads: 2, r_max: 8}7init: {freeze_backbone: false}8data: {kind: synthetic, tokenizer: char, splits_dir: data/splits/synthetic_v1, loss_span: all,9       eval_splits: [dev_iid, dev_template, dev_depth, dev_compose]}10train: {batch_size: 64, seq_len: 320, steps: 6000, lr: 1.0e-3, warmup: 200, weight_decay: 0.1, grad_clip: 1.0,11        log_every: 50, eval_every: 500, eval_batches: 10, ckpt_every: 2000}12em_max_items: 60013
Files · main
Menu

Theme