spb/localvm-research Public License

1---2project: localvm-research3document: expA_weight_concentration/analysis4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6created: 2026-08-127status: reviewed8---910# Analysis — expA_weight_concentration1112Run: `results/expA_weight_concentration/20260812T044829Z/` · Qwen3-1.7B bf16,1348 trajectories × 128 tokens (6 144 positions), SwiGLU intermediate energy14per 16-neuron block (384 blocks × 28 layers), 64-neuron granularity derived.1516```text17Hypothesis / Falsification18  Hoped: top 20% of 64-neuron blocks ≥60% energy; ≤50% of blocks for 95%.19  Kill: >70% of 64-neuron blocks needed for 95% energy.2021Result — KILL CRITERION TRIGGERED at 64-neuron granularity22  Granularity   top10%   top20%   needed for 90% / 95% / 99%23  neuron        0.88     0.94     0.13 / 0.20 / 0.3824  block-16      0.58     0.71     0.48 / 0.61 / 0.8125  block-64      0.44     0.57     0.65 / 0.77 / 0.92   ← 0.77 > 0.70 kill line26  Domain-independent (95% needs 61–62% of block-16 across all six domains).27  Strong depth gradient: late layers concentrate (L27 needs 11% of blocks28  for 95%; L20–26: 30–47%) while early/mid layers are diffuse (64–81%).2930Interpretation31  1. NEGATIVE at pageable granularity: real per-token concentration exists32     at neuron level (20% for 95% — consistent with TEAL-class ~40–50%33     approximate sparsity claims), but bundling to SSD-friendly blocks34     destroys it: at 64 neurons (≈ 256 KiB rows bundle at 1.7B dims) the35     important set is 77% of the layer — no meaningful byte savings.36     Important neurons are SCATTERED, not clustered: block energy ≈37     uniform mixing. This is the quantitative reason LLM-in-a-flash used38     ReLU models — SwiGLU energy has no exploitable block structure.39  2. The expH fetch contract (≥256 KiB) and neuron-level concentration40     (4 KiB-scale rows) are mutually exclusive on this architecture:41     the SSD wants big blocks, the sparsity lives in small ones. A42     permutation/clustering pass (grouping co-active neurons) is the one43     remaining idea for this route — but expB (below) must first show the44     sets are stable enough to be worth clustering.45  3. The depth gradient is scientifically interesting: late layers are46     energy-concentrated but (expF) decision-insensitive; early layers are47     decision-relevant but energy-diffuse. Energy is not the right48     importance signal for escalation — margins are (expG).4950Next experiment51  expB on the same trace (temporal stability) — run before drawing final52  conclusions on route (c); if sets churn too, the sparsity-paging family53  (G06/G07/G12/G13) dies for dense SwiGLU models at this scale.54```55