SPB Git

spb/wp31_uqo

Public
3.5 KB · 49 lines markdown
1# PRISME v1 — frozen architecture specification (Phase 1, 2026-10-29)23Notation: prefix `x_{1:t}`, backbone residual stream `H_t = B_phi(x_{1:t}) ∈ R^{t×d}`, `h_t = H_t[t]`.4Workspace width `w`, `M = 16` slots, editor blocks `L_w = 2` (shared weights), hard cap `R_max = 8`.56## Backbone (`prisme/backbone.py`)7Pre-norm causal decoder: RMSNorm → GQA attention (RoPE θ = 10⁴, head dim 64) → RMSNorm → SwiGLU; tied8input/output embeddings; one final RMSNorm. `logits = E · RMSNorm(h)`. Exact backbone counts match the brief9for the three configurations (`results/tables/param_counts.json`).1011## Source bank and workspace initialisation (`prisme/model.py`)12`ĥ = RMSNorm_bank(H)` (a dedicated norm, d parameters, added for numerical hygiene — see DECISIONS D2),13`K = ĥ W_K`, `V = ĥ W_V` (read-only, causal: query t reads keys 1..t only), `z_t = P ĥ_t`,14`S0[j] = e[j] + z_t` for j = 0..15; S is re-initialised at every position (no cross-token memory).1516## Action catalog (`prisme/workspace.py`)17`SINGLE_0..15`, `PAIR_0_1, PAIR_2_3, …, PAIR_14_15`, `STOP` = 25 actions (24 nonterminal). STOP is absorbing.1819## Shared sparse editor (`prisme/editor.py`)20Active queries `x = S[I(a)] + ctx_proj([step/R, remaining/R, k/2])`. Each of the two shared blocks:21(1) RMSNorm → multi-head cross-attention of the active queries to the bank (q, o projections only; the bank is22already in width w) with the strict per-query causal mask; (2) RMSNorm → multi-head attention of the active23queries over all M slots of the **pre-write snapshot** (same norm applied to the snapshot); (3) RMSNorm → SwiGLU.24Write: `δ = proposed − S[j]`, `δ_c = δ · min(1, c / max(RMS(δ), ε))`, `η = η_max σ(gate([S[j]; proposed]))`,25`S'[j] = S[j] + η δ_c`. Non-selected slots are returned bitwise unchanged (one-hot scatter from the same26snapshot ⇒ pair writes are order-independent). Defaults: η_max = 0.25, c = 1 (untested research values).2728## Value controller (`prisme/controller.py`)29Target-free features: `u = f(z_t, step, remaining)` (one SwiGLU MLP per state), per action30`h_a = SiLU(u + W_s s_a + k_emb[k(a)])` with `s_a` the mean of the slots in I(a); heads → `ĝ, ĉ, log σ̂, harm`.31`Q(s,a) = ĝ − λ c(a) − κ σ̂`, `Q(STOP) = 0`, greedy argmax, STOP when max Q ≤ 0. `c(a)` is the analytic32ops cost of the action relative to the backbone ops/token (`prisme/flops.py`), not a learned quantity.3334## Differential readout and exact bypass (`prisme/readout.py`, `prisme/model.py`)35`r(S, z) = Σ_j softmax_j((W_q z)·(W_k S_j)/√w) S_j`; `Δ = U (r(S_final, z) − r(S0, z))`, U = 0 at init;36`h_corr = h + Δ`. `max_budget = 0` ⇒ backbone head returned without allocating slots (explicit branch);37positions with zero executed edits also bypass the readout. Identity is bitwise (tests 02–04).3839## Causal / cache contract (`prisme/cache.py`, `prisme/generation.py`)40Separate per-request caches for backbone KV and bank K/V; the editor never writes to H, K, V, z; incremental41decoding equals full-prefix recomputation within 1e-4 (tests in `tests/integration/`).4243## Parameter accounting44Implemented workspace+controller: 2,631,173 (103M) / 10,505,221 (459M) / 21,655,813 (1.18B) versus the brief's453,088,260 / 12,336,900 / 25,779,844. The brief's three numbers are reproduced exactly by46`4dw + Mw + 17w² + 6w f_w + 15.5w + 4`; the implementation has `10w²` square projections instead of `17w²`47(no K/V re-projection of the bank inside the editor, no value projection in the readout, leaner controller)48and a few extra vectors (bank norm, context projections, k-embedding). See DECISIONS D3.49
Files · a2c46f03643fdd75e8e87595931d35e18c590d57
Menu

Theme