project: localvm-research document: expA_weight_concentration/hypothesis author: Simon-Pierre Boucher contact: contact@spboucher.ai created: 2026-08-12 modified: 2026-08-12 status: reviewed
Hypothesis — expA_weight_concentration
After expF killed layer-granularity escalation, this measures the next grain down: are FFN weight blocks (bundled neurons) unequally important per token? Feeds G01/G06/G07 and expE (partial GEMM); shares its trace with expB.
text
Hypothesis
Per-token FFN intermediate-activation energy is concentrated: on a modern
SwiGLU model, the top 20% of 64-neuron blocks capture ≥60% of the energy,
and ≤50% of blocks suffice for 95% of the energy (per token, averaged
across positions and domains). Per-neuron concentration is substantially
stronger than block-64 concentration (bundling cost is real but moderate).
Falsification criterion
If capturing 95% of per-token energy requires >70% of 64-neuron blocks
(near-uniform importance), then block-level weight selection cannot cut
bytes materially on this architecture and G06-style paging must rely on
thresholded sparsity of individual neurons or die; C1 escalation-byte
reduction via block selection (route c from expF) is dead too.
Method
Qwen3-1.7B bf16. Wrap every layer's mlp.down_proj with a recorder; its
input IS the SwiGLU intermediate activation h = silu(gate(x))·up(x),
whose per-neuron magnitude determines the contribution of up/gate rows
and down columns (the Gate-Up-Down bundle of the paging literature).
Forward the 48 reference trajectories (same protocol as expG/D/F,
greedy 128-token continuations, teacher-forced positions only).
Record per predicted position: block energy (sum of h² over 64-neuron
blocks; 96 blocks × 28 layers), stored float16 npz for expB reuse; plus
streaming per-neuron stats (fraction of neurons for 90/95/99% energy).
Report: energy captured by top {10,20,40,60}% blocks; blocks needed for
{90,95,99}% energy; per-layer, per-domain aggregates; neuron-vs-block
comparison.
Baseline
Uniform importance (top k% of blocks capture exactly k% of energy) —
the null hypothesis; and per-neuron granularity as the upper bound on
achievable concentration.
Result
KILL TRIGGERED at 64-neuron granularity: 95% energy needs 77% of blocks (>70% line). Neuron-level real (20% for 95%) but scattered — bundling destroys it. Depth gradient: late layers concentrated, early diffuse. Domain-independent.
Interpretation
SwiGLU energy has no exploitable block structure; SSD fetch contract (≥256 KiB) and neuron-scale sparsity are mutually exclusive. Energy ≠ decision importance (cf. expF).
Next experiment
expB on the same trace; then pivot decision.