---
project: localvm-research
document: expA_weight_concentration/hypothesis
author: Simon-Pierre Boucher
contact: contact@spboucher.ai
created: 2026-08-12
modified: 2026-08-12
status: reviewed
---

# Hypothesis — expA_weight_concentration

After expF killed layer-granularity escalation, this measures the next grain
down: are FFN weight *blocks* (bundled neurons) unequally important per token?
Feeds G01/G06/G07 and expE (partial GEMM); shares its trace with expB.

```text
Hypothesis
  Per-token FFN intermediate-activation energy is concentrated: on a modern
  SwiGLU model, the top 20% of 64-neuron blocks capture ≥60% of the energy,
  and ≤50% of blocks suffice for 95% of the energy (per token, averaged
  across positions and domains). Per-neuron concentration is substantially
  stronger than block-64 concentration (bundling cost is real but moderate).

Falsification criterion
  If capturing 95% of per-token energy requires >70% of 64-neuron blocks
  (near-uniform importance), then block-level weight selection cannot cut
  bytes materially on this architecture and G06-style paging must rely on
  thresholded sparsity of individual neurons or die; C1 escalation-byte
  reduction via block selection (route c from expF) is dead too.

Method
  Qwen3-1.7B bf16. Wrap every layer's mlp.down_proj with a recorder; its
  input IS the SwiGLU intermediate activation h = silu(gate(x))·up(x),
  whose per-neuron magnitude determines the contribution of up/gate rows
  and down columns (the Gate-Up-Down bundle of the paging literature).
  Forward the 48 reference trajectories (same protocol as expG/D/F,
  greedy 128-token continuations, teacher-forced positions only).
  Record per predicted position: block energy (sum of h² over 64-neuron
  blocks; 96 blocks × 28 layers), stored float16 npz for expB reuse; plus
  streaming per-neuron stats (fraction of neurons for 90/95/99% energy).
  Report: energy captured by top {10,20,40,60}% blocks; blocks needed for
  {90,95,99}% energy; per-layer, per-domain aggregates; neuron-vs-block
  comparison.

Baseline
  Uniform importance (top k% of blocks capture exactly k% of energy) —
  the null hypothesis; and per-neuron granularity as the upper bound on
  achievable concentration.

Result
  KILL TRIGGERED at 64-neuron granularity: 95% energy needs 77% of blocks (>70% line). Neuron-level real (20% for 95%) but scattered — bundling destroys it. Depth gradient: late layers concentrated, early diffuse. Domain-independent.

Interpretation
  SwiGLU energy has no exploitable block structure; SSD fetch contract (≥256 KiB) and neuron-scale sparsity are mutually exclusive. Energy ≠ decision importance (cf. expF).

Next experiment
  expB on the same trace; then pivot decision.
```