---
project: localvm-research
document: expA_weight_concentration/analysis
author: Simon-Pierre Boucher
contact: contact@spboucher.ai
created: 2026-08-12
status: reviewed
---

# Analysis — expA_weight_concentration

Run: `results/expA_weight_concentration/20260812T044829Z/` · Qwen3-1.7B bf16,
48 trajectories × 128 tokens (6 144 positions), SwiGLU intermediate energy
per 16-neuron block (384 blocks × 28 layers), 64-neuron granularity derived.

```text
Hypothesis / Falsification
  Hoped: top 20% of 64-neuron blocks ≥60% energy; ≤50% of blocks for 95%.
  Kill: >70% of 64-neuron blocks needed for 95% energy.

Result — KILL CRITERION TRIGGERED at 64-neuron granularity
  Granularity   top10%   top20%   needed for 90% / 95% / 99%
  neuron        0.88     0.94     0.13 / 0.20 / 0.38
  block-16      0.58     0.71     0.48 / 0.61 / 0.81
  block-64      0.44     0.57     0.65 / 0.77 / 0.92   ← 0.77 > 0.70 kill line
  Domain-independent (95% needs 61–62% of block-16 across all six domains).
  Strong depth gradient: late layers concentrate (L27 needs 11% of blocks
  for 95%; L20–26: 30–47%) while early/mid layers are diffuse (64–81%).

Interpretation
  1. NEGATIVE at pageable granularity: real per-token concentration exists
     at neuron level (20% for 95% — consistent with TEAL-class ~40–50%
     approximate sparsity claims), but bundling to SSD-friendly blocks
     destroys it: at 64 neurons (≈ 256 KiB rows bundle at 1.7B dims) the
     important set is 77% of the layer — no meaningful byte savings.
     Important neurons are SCATTERED, not clustered: block energy ≈
     uniform mixing. This is the quantitative reason LLM-in-a-flash used
     ReLU models — SwiGLU energy has no exploitable block structure.
  2. The expH fetch contract (≥256 KiB) and neuron-level concentration
     (4 KiB-scale rows) are mutually exclusive on this architecture:
     the SSD wants big blocks, the sparsity lives in small ones. A
     permutation/clustering pass (grouping co-active neurons) is the one
     remaining idea for this route — but expB (below) must first show the
     sets are stable enough to be worth clustering.
  3. The depth gradient is scientifically interesting: late layers are
     energy-concentrated but (expF) decision-insensitive; early layers are
     decision-relevant but energy-diffuse. Energy is not the right
     importance signal for escalation — margins are (expG).

Next experiment
  expB on the same trace (temporal stability) — run before drawing final
  conclusions on route (c); if sets churn too, the sparsity-paging family
  (G06/G07/G12/G13) dies for dense SwiGLU models at this scale.
```