spb/localvm-research Public License

1---2project: localvm-research3document: expA_weight_concentration/hypothesis4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6created: 2026-08-127modified: 2026-08-128status: reviewed9---1011# Hypothesis — expA_weight_concentration1213After expF killed layer-granularity escalation, this measures the next grain14down: are FFN weight *blocks* (bundled neurons) unequally important per token?15Feeds G01/G06/G07 and expE (partial GEMM); shares its trace with expB.1617```text18Hypothesis19  Per-token FFN intermediate-activation energy is concentrated: on a modern20  SwiGLU model, the top 20% of 64-neuron blocks capture ≥60% of the energy,21  and ≤50% of blocks suffice for 95% of the energy (per token, averaged22  across positions and domains). Per-neuron concentration is substantially23  stronger than block-64 concentration (bundling cost is real but moderate).2425Falsification criterion26  If capturing 95% of per-token energy requires >70% of 64-neuron blocks27  (near-uniform importance), then block-level weight selection cannot cut28  bytes materially on this architecture and G06-style paging must rely on29  thresholded sparsity of individual neurons or die; C1 escalation-byte30  reduction via block selection (route c from expF) is dead too.3132Method33  Qwen3-1.7B bf16. Wrap every layer's mlp.down_proj with a recorder; its34  input IS the SwiGLU intermediate activation h = silu(gate(x))·up(x),35  whose per-neuron magnitude determines the contribution of up/gate rows36  and down columns (the Gate-Up-Down bundle of the paging literature).37  Forward the 48 reference trajectories (same protocol as expG/D/F,38  greedy 128-token continuations, teacher-forced positions only).39  Record per predicted position: block energy (sum of h² over 64-neuron40  blocks; 96 blocks × 28 layers), stored float16 npz for expB reuse; plus41  streaming per-neuron stats (fraction of neurons for 90/95/99% energy).42  Report: energy captured by top {10,20,40,60}% blocks; blocks needed for43  {90,95,99}% energy; per-layer, per-domain aggregates; neuron-vs-block44  comparison.4546Baseline47  Uniform importance (top k% of blocks capture exactly k% of energy) —48  the null hypothesis; and per-neuron granularity as the upper bound on49  achievable concentration.5051Result52  KILL TRIGGERED at 64-neuron granularity: 95% energy needs 77% of blocks (>70% line). Neuron-level real (20% for 95%) but scattered — bundling destroys it. Depth gradient: late layers concentrated, early diffuse. Domain-independent.5354Interpretation55  SwiGLU energy has no exploitable block structure; SSD fetch contract (≥256 KiB) and neuron-scale sparsity are mutually exclusive. Energy ≠ decision importance (cf. expF).5657Next experiment58  expB on the same trace; then pivot decision.59```60