spb/localvm-research Public License

1---2project: localvm-research3document: research/notes/sparsity_pruning4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6created: 2026-08-117status: draft8---910# Activation Sparsity & Weight Pruning — Research Notes (charter §4.2, §4.3)1112Deep literature scan, 2022–2026. Focus: which sparsity/pruning mechanisms decouple13`total model size` from `resident model size` and `bytes read per token`, and which of them14actually map to Apple Silicon unified memory + Metal + internal NVMe.1516---1718## 1. Landscape overview1920Two families, one shared premise:2122- **Activation sparsity (§4.2):** for a given input, only a small input-dependent subset of23  FFN neurons / attention heads meaningfully contributes to the output. If that subset can be24  *predicted cheaply before the matmul*, the untouched weights never need to be read —25  a **bandwidth** and (with an offload tier) a **capacity** win, with no permanent model change.26- **Weight pruning (§4.3):** some weights contribute little for *all* inputs and can be removed27  permanently (one-shot unstructured: SparseGPT, Wanda; structured: SliceGPT, LLM-Pruner,28  Sheared-LLaMA, Minitron; depth: ShortGPT). A **capacity** win, but paid for with permanent29  quality loss and (for unstructured) very poor hardware realizability.3031Key historical arc:32331. **2022–2023 (ReLU era):** the "lazy neuron" observation — in ReLU transformers <10% of FFN34   neurons fire per token (3.0% for T5-Base). DejaVu turns this into *contextual sparsity* with35   per-layer predictors (up to 85% sparsity, 2× over FasterTransformer on OPT-175B).362. **2023–2024 (offload era):** PowerInfer (hot/cold neurons split across GPU/CPU),37   **LLM in a flash** (Apple: neurons paged from flash on demand), PowerInfer-2 / Ripple38   (smartphones, UFS storage). Sparsity becomes a **cache/paging policy**, not just a FLOP saver.39   This is the closest prior art to the localvm-research goal.403. **2024–2026 (SwiGLU problem & training-free era):** modern models (Llama-2/3, Mistral, Qwen)41   use SwiGLU — activations are no longer exactly zero. Responses: *ReLUfication* retraining42   (ReLU Strikes Back, ProSparse, TurboSparse dReLU, Q-Sparse), or *training-free thresholding*43   of near-zero activations (CATS, TEAL, GRIFFIN, DIP, R-Sparse, SparseInfer, statistical44   calibration). Sirius (NeurIPS 2024) shows the quality cost is concentrated in reasoning tasks45   and is *recoverable by correction*.464. **Unresolved:** essentially all fast implementations are CUDA (Triton kernels, GPU/CPU split)47   or Android/UFS. **No system implements predictor-driven sparse weight paging on48   macOS/Metal/Apple NVMe.** PowerInfer's own README lists "Metal backend for sparse inference49   on macOS" as *planned, not implemented* (macOS today = CPU only, "limited" gains).5051The most important conceptual distinction for us: pruning research asks *"which weights can we52delete?"*; offload-sparsity research asks *"which weights must be resident right now?"* The53second question converts pruning from a destructive transform into a **cache-residency policy**54— weights are demoted to NVMe, not destroyed, and recovered when the input distribution needs55them. Only a handful of systems (LLM in a flash, PowerInfer-2, M2Cache, DIP, Ripple) partially56do this, none on a Mac.5758---5960## 2. Techniques and systems6162### 2.1 DejaVu — contextual sparsity (ICML 2023 oral)6364- **Mechanism.** Hypothesis: small, input-dependent sets of attention heads and FFN neurons65  reproduce the dense output for each input. Trains small MLP predictors per layer; exploits66  slowly-changing hidden states across layers to predict layer ℓ's sparsity from layer ℓ−1's67  input (asynchronous "lookahead" hides predictor latency).68- **Memory / bandwidth.** No resident-memory reduction (full model stays in GPU HBM). Bandwidth:69  up to 85% contextual sparsity ⇒ proportionally fewer weight bytes streamed from HBM per token.70  Contextual sparsity has up to 7× better efficiency–accuracy trade-off than static sparsity.71- **Quality.** "Without compromising model quality" on OPT-175B; >2× latency reduction vs72  FasterTransformer, >6× vs HuggingFace.73- **Predictor overhead.** Small per-layer MLPs; hidden by async lookahead on A100s.74- **Hardware assumptions.** OPT (ReLU FFN), multi-A100 discrete-GPU serving. The75  GPU-resident-model assumption gives it no capacity benefit; irrelevant as-is for a76  memory-constrained Mac, but the *predictability result* (85%) is foundational.77- **Limitation.** ReLU-dependent; whole model must fit in accelerator memory; predictors trained78  per model.79- **Extension for us.** Reuse the "slowly changing hidden states" property as a *prefetch signal*80  for NVMe reads rather than a FLOP-skip signal.8182### 2.2 PowerInfer — hot/cold neuron split on one consumer GPU (SOSP 2024)8384- **Mechanism.** Neuron activations follow a **power-law**: a small "hot" set is activated across85  almost all inputs, the "cold" majority is input-specific. Hot neurons preloaded to GPU VRAM;86  cold neurons computed on CPU; adaptive per-layer predictors + neuron-aware sparse operators.87- **Numbers.** OPT-175B-class models on one RTX 4090: 13.2 tok/s average, 29.08 peak — only 18%88  below an A100. Requires ReLU-family models (ReluLLaMA, ProSparse-LLaMA, Bamboo, TurboSparse);89  explicitly does **not** support vanilla Llama/Mistral/Qwen.90- **Memory / bandwidth.** GPU-resident set ≪ model size (capacity win via CPU DRAM as second91  tier); bandwidth win from computing only predicted-active neurons.92- **Hardware assumptions.** **Discrete GPU + PCIe + separate CPU DRAM.** This split is93  *meaningless on Apple unified memory* — there is no "GPU VRAM vs CPU RAM" distinction; the Mac94  analogue is RAM (hot) vs NVMe (cold), i.e., exactly the LLM-in-a-flash setting.95- **macOS status (verified on repo).** Runs on Apple M chips CPU-only with "limited" improvement;96  Metal sparse backend is a *planned feature that never shipped*. Project still active97  (SmallThinker 2025, Tiiny AI Pocket Lab Jan 2026) but the Mac gap remains.98- **Extension.** Port the hot/cold *statistical* insight to a RAM/NVMe hierarchy with Metal99  gather-matvec kernels; hot set pinned in wired memory, cold set demand-paged.100101### 2.3 LLM in a flash (Apple, ACL 2024) — deepest dive, closest prior art #1102103*Alizadeh et al., arXiv 2312.11514. The only major paper that ran sparse weight paging on actual104Apple hardware.*105106**What exactly they did:**107108- **Hardware:** Apple **M1 Max** (1TB SSD) and **M2 Ultra** (2TB SSD), CPU (fp32) and Metal GPU109  (fp16) paths; plus Linux RTX 4090 (bf16). Memory budget: ~**half the model size** in DRAM.110- **Models:** OPT-6.7B, sparsified/ReLUfied Falcon-7B, Persimmon-8B, Phi-2, FATReLU Llama-2-7B.111  All ReLU-family FFNs.112- **Predictor:** low-rank predictor per FFN layer (OPT-6.7B: rank 128 for layers 1–28, rank 1024113  for last 4). Trained on 10k C4 samples, 2 epochs, ~4 h/predictor on A100. Cost: <2.4% of114  non-embedding weights/FLOPs; ~5% false negatives, 7% false positives; 2.75% (CPU) / 4.8% (GPU)115  of compute time.116- **Windowing:** keep the union of neurons activated in the last k=4–5 tokens in DRAM; per new117  token only load the *delta*. OPT-6.7B at k=4: each token touches **2.4% of FFN neurons**;118  FFN occupies only 15.5% of DRAM; total DRAM ≈ **52.1% of model size**. Falcon-7B: 3.1% of FFN119  neurons/token, DRAM ≈ 52.9%.120- **Row–column bundling:** store up-projection column i together with down-projection row i so121  one neuron = one contiguous 2·d_model read → doubles chunk size. Measured effective flash122  throughput: ~1.25 GiB/s (predictor+windowing) → **~2.25 GiB/s with bundling**, vs >6 GiB/s123  sequential on M1 Max. Sweet spot: ≥32 KiB random reads across **32 threads**.124- **Bytes/token (OPT-6.7B):** naive 13.4 GB → predictor only 6.7 GB → **predictor+windowing125  0.2 GB per token**. (This is the single most important measured number for our charter §8.3.)126- **Speedups:** OPT-6.7B total per-token latency 669 ms CPU (4.75× vs naive), 565 ms Metal/M1127  (4.23×), 305 ms Metal/M2 Ultra (7.44×), 84 ms CUDA (26.4×); +speculative → 37×. Runs models128  up to **2× DRAM size**.129130**What they did NOT do (our opening):**131132- Attention weights + embeddings kept **permanently in DRAM** (~19–32% of model) — only FFN133  weights are paged. No paging of attention, no KV-cache tiering.134- Predicated on **ReLU sparsity**; dense SwiGLU models "not addressed" (their words: the method135  "is constructed on the foundation of sparsified networks").136- Only ~2× DRAM oversubscription demonstrated on ≤8B models — not 4–10×, not 30–70B on a Mac.137- Single-batch, greedy decoding only; no power measurement; no quantization co-design (fp16138  neurons on flash — 4-bit bundles would quadruple effective neuron throughput).139- **No code release.** Nothing in MLX/llama.cpp implements it today.140- Negative result they report: bundling by *co-activation* (closest coactivated neighbor) failed141  — hot neurons got loaded repeatedly. (Ripple later solved this with global placement.)142143### 2.4 PowerInfer-2 — deepest dive, closest prior art #2144145*Xue et al., arXiv 2406.06282. First 47B model on a smartphone.*146147**What exactly they did:**148149- **Hardware:** OnePlus 12 (24 GB DRAM, 19 GB usable, UFS 4.0, Snapdragon 8 Gen 3) and OnePlus150  Ace 2 (16 GB, UFS 3.1). UFS 4.0: ~4 GB/s sequential (512 KB), **~1 GB/s at 4 KB random**,151  850 MB/s over larger ranges.152- **Models:** TurboSparse-Mixtral-47B (dReLU MoE, ~3B activated params/token),153  TurboSparse-Mistral-7B, Bamboo-7B, sparse Llama-13B, Qwen2-7B; SiLU Mistral-7B as a stress154  case.155- **Neuron cluster abstraction:** groups of same-layer FFN neurons with similar activation156  statistics. Hot (frequently active) clusters → large, dense, NPU-computed; cold clusters →157  small, sparse, CPU-computed. Prefill: NPU dense matmuls while a CPU core streams weights.158  Decode: NPU handles ~70% (hot dense part), CPU the sparse remainder.159- **Segmented neuron cache:** three DRAM regions — (1) pinned attention + KV, (2) hot region160  (cluster-granularity LRU), (3) cold region (neuron-granularity LRU); ratios adapt to batch161  size.162- **I/O pipeline:** neuron-cluster-level pipelining overlaps compute on cached clusters with UFS163  reads of missing ones. Cold neurons stored as **Gate-Up-Down bundles** (~80% co-activation164  across the three matrices). For 4-bit models: two-phase read — 4 KB Gate first, Up/Down 4 KB165  fetched *only if* Gate output ≠ 0.166- **Results:** 11.68 tok/s decoding for TurboSparse-Mixtral-47B on 24 GB phone (up to 27.8–29×167  vs llama.cpp; 3.84× vs "LLMFlash" i.e. LLM-in-a-flash-style baseline). With ~50% FFN offload168  on a 7B: ~11.1 tok/s vs ~14.5 in-memory — offload costs only ~23%.169- **Overhead accounting (7 GB budget, 47B model):** 1 GB non-FFN weights + **2.6 GB predictors**170  + 2.7 GB quantization scales + 0.3 GB runtime = 6.6 GB, leaving only 400 MB of neuron cache171  (1.8% of FFN weights) — still runs. Predictor DRAM cost is *large*, a real design lesson.172173**What they did NOT do:**174175- Rooted **Android only**; "iOS portability" claimed but never validated. Nothing on176  macOS/Metal/ANE.177- Depends on **dReLU ReLUfied models** (TurboSparse = SFT-retrained Mistral/Mixtral, ~150B178  tokens); on stock SiLU models speedup drops to 2.4× (vs 4.6× ReLU). Not post-training-only.179- Offline per-device planner required; no cross-device generalization; high tail latency180  (P99 +40.9% vs mean).181- Never tested Apple-class NVMe (6+ GB/s vs their 1 GB/s random UFS): **the Mac's storage is182  ~4–6× faster than the storage this system was designed around** — its economics should183  transfer favorably, but nobody has built it.184185### 2.5 Ripple / Neuralink — co-activation-aware flash layout (2024)186187- **Mechanism.** Neurons that fire together are placed together in flash ("neuron co-activation188  linking"), converting many small random reads into fewer large sequential reads — directly189  attacking the IOPS bound that limits smartphone (and, less severely, Mac) storage.190- **Relevance.** Solves exactly the negative result LLM in a flash reported for co-activation191  bundling, via offline global placement optimization. An offline "layout compiler" of this192  kind belongs in our compile stage (§14 of charter). Hardware: Android/UFS; no Mac port.193194### 2.6 ShadowLLM — better importance predictors (EMNLP 2024)195196- **Mechanism.** Single early-layer predictor "shadows" the whole model instead of per-layer197  predictors; goes beyond magnitude-based criteria for head/neuron importance.198- **Numbers.** >15% end-to-end accuracy improvement over DejaVu-style criteria at equal sparsity;199  up to 20% speedup over DejaVu; validated on OPT/Llama-2 up to 30B. Code: abdelfattah-lab/shadow_llm.200- **Hardware.** CUDA. Predictor design is transferable; a single-point predictor is attractive on201  Mac because it gives *maximum I/O prefetch lead time* (predict at layer 0 → prefetch layer 30).202203### 2.7 ReLUfication line: ReLU Strikes Back → ProSparse → TurboSparse → Q-Sparse204205- **ReLU Strikes Back (Apple, ICLR 2024):** swapping SiLU/GELU→ReLU and fine-tuning has206  negligible quality impact while enabling up to ~3× less computation/weight transfer at the207  memory-bound decode step. Establishes that sparsity is *recoverable post-training* with modest208  fine-tuning.209- **ProSparse (2402.13516):** activation substitution + progressive L1 regularization + threshold210  shifting → **89.3% / 88.8% / 87.9%** activation sparsity on LLaMA2-7B/13B/MiniCPM-1B at parity211  with the Swish originals.212- **TurboSparse (2406.05955):** **dReLU** + data-mix continued training (~150B tokens) →213  Mistral-7B activates only **2.5B** params/token; Mixtral-47B activates **4.3B**; 2–5× decode214  speedup; 11 tok/s on phones (feeds PowerInfer-2).215- **Q-Sparse (2407.10969, NeurIPS 2024):** top-K activation sparsification with STE during216  training; full activation sparsity, inference-optimal scaling law for sparse LLMs; works with217  BitNet 1.58-bit. Training-time method — violates our post-training-only constraint but maps218  the ceiling.219- **Caveat for us:** all of these need GPU-scale fine-tuning (out of scope to *produce*, but the220  checkpoints — ProSparse-LLaMA, Bamboo-7B, TurboSparse-Mistral/Mixtral — are downloadable and221  are ideal *test vehicles* on the Mac).222223### 2.8 Training-free SwiGLU sparsity: CATS, TEAL, GRIFFIN, DIP, and successors224225- **CATS (COLM 2024):** thresholds the **gate output** of SwiGLU blocks (per-layer thresholds226  from calibration distributions). ~**99% of base task performance at 50% FFN activation227  sparsity** on Mistral-7B/Llama2-7B without fine-tuning; but only sparsifies Wup/Wdown ⇒228  ~25% model-wide sparsity. Custom GPU kernel gives wall-clock gains.229- **TEAL (ICLR 2025):** magnitude-thresholds **hidden states model-wide** (every matrix incl.230  attention), exploiting zero-mean unimodal activation distributions. **40–50% model-wide231  sparsity with minimal degradation** on Llama-2/-3/Mistral 7B–70B; 1.53×/1.8× decode speedup at232  40/50% via Triton gather kernels; composes with weight quantization. **CUDA-only kernels** —233  the natural first Metal port target.234- **GRIFFIN (ICML 2024):** "flocking" — within a sequence, tokens activate largely the *same* FF235  neurons. Selects experts once per sequence from the prompt, no training/calibration, works on236  many non-ReLU activations. **50% of FF parameters with little-to-no degradation** (generation +237  classification), lower latency. Sequence-level selection = coarse, *prefetch-friendly*238  granularity (one NVMe read burst per sequence, not per token).239- **DIP — Dynamic Input Pruning with Cache-Aware Masking (Qualcomm, 2412.01380):**240  predictor-free magnitude sparsification of SwiGLU + optional LoRA recovery + **cache-aware241  masking**: the sparsity mask is chosen considering *what is already in the DRAM cache*,242  trading tiny accuracy for large cache-hit-rate gains. Phi-3-Medium under mobile DRAM limits:243  **46% less memory, 40% more throughput, <0.1 ppl loss** vs streaming the dense model. This is244  the first explicit "sparsity-as-cache-policy" formulation — conceptually the closest paper to245  our §4.3 key question, but on simulated mobile constraints, CUDA/simulation, no Mac.246- **Others (2024–2026):** SparseInfer (training-free sign-bit activation prediction);247  Post-Training Statistical Calibration (2412.07174); R-Sparse (rank-aware, training-free,248  attention+FFN); La RoSA (layerwise rotation before sparsification); Spark Transformer249  (Google, NeurIPS 2025: restores FFN+attention sparsity with low-cost top-k predictor);250  CETT/“Universal Properties” (2509.00454: sparsity potential *grows with model size*, first251  diffusion-LLM study); Fast Forward (2602.00397: predictive FFN sparsity for *prefill*);252  tree-structured FFN dynamic sparsity at scale (2604.08565); flexible N:M *activation*253  sparsity benchmarking for next-gen accelerators (2509.22166).254255### 2.9 SparQ Attention — the attention-side analogue (ICML 2024)256257- Fetches only the KV-cache rows whose keys matter for the current query (rank the query's large258  components, approximate scores, fetch top keys). **Up to 8× reduction in attention data259  transfer** with negligible loss on Llama-2/3, Mistral, Gemma, Pythia; no fine-tuning.260- Relevance: our working-set question applies to KV as well as weights; SparQ shows261  demand-driven fetching works for attention state. Complements FFN-side sparsity (weights262  dominate at short context, KV at long context).263264### 2.10 One-shot weight pruning: SparseGPT & Wanda (verified numbers)265266- **SparseGPT (ICML 2023):** layer-wise sparse regression with approximate Hessian-based weight267  updates; prunes OPT-175B/BLOOM-176B in <4.5 h on one GPU; 50–60% unstructured with small ppl268  increase at 100B+ scale.269- **Wanda (ICLR 2024):** score = |weight| × ‖input activation‖, per-output-row, no weight update;270  ~300× faster to compute than SparseGPT.271- **WikiText-2 perplexity (dense → magnitude / SparseGPT / Wanda at 50% unstructured):**272  - LLaMA-7B: 5.68 → 17.29 / 7.22 / 7.26273  - LLaMA-13B: 5.09 → 20.21 / 6.21 / 6.15274  - LLaMA-65B: 3.56 → 5.90 / 4.57 / 4.57275  - LLaMA-2-70B: 3.12 → 4.98 / 3.98 / 3.98276  - 2:4 structured is much worse at small scale (LLaMA-2-7B: 5.12 → 11.02 Wanda).277- **Interpretation:** (i) one-shot 50% is nearly free only at ≥65B scale; at 7B it costs278  ~25–28% ppl. (ii) Hardware-friendly 2:4 patterns are the most damaging. (iii) End-to-end gain279  is meager even on NVIDIA (Wanda reports 1.24× e2e with 2:4 on LLaMA-7B) — and **Apple GPUs280  have no sparse tensor cores at all**, so unstructured weight sparsity yields *zero* bandwidth281  savings on a Mac unless the representation itself skips loads (which CSR-style indexing282  overhead largely cancels — see Endor).283- **Reframing for us:** Wanda's per-input-statistics score is cheap enough to compute *online*;284  it can rank weights for **residency** (RAM vs NVMe) instead of deletion.285286### 2.11 Structured / depth / width pruning: SliceGPT, LLM-Pruner, ShortGPT, Sheared-LLaMA, Minitron287288- **SliceGPT (ICLR 2024):** orthogonal-rotation + slicing; removes up to 25% of parameters with289  99% (Llama-2-70B, OPT-66B) / 90% (Phi-2) zero-shot retention; produces smaller *dense*290  matrices — the only pruning family that trivially runs fast on Metal (it's just smaller GEMMs).291- **LLM-Pruner (NeurIPS 2023):** gradient-based structural pruning + LoRA recovery; ~20%292  compression at moderate loss.293- **ShortGPT (2403.03853):** Block Influence = 1 − cos-sim(layer input, output); middle-to-late294  layers (e.g. 21–29 of LLaMA-2-7B) are highly redundant; deleting whole layers beats fancier295  methods on many benchmarks, though generation/reasoning suffer more than classification296  (follow-ups: Prune&Comp, E³-Pruner, layer-pruning-limits studies 2025–26).297- **Sheared-LLaMA (ICLR 2024):** Lagrangian-learned masks over layers/heads/dims + dynamic batch298  loading; LLaMA2-7B → 1.3B/2.7B using ~50B tokens (3% of from-scratch compute); beats equal-size299  models trained from scratch.300- **Minitron (NVIDIA 2024):** iterative width(+depth) pruning + KD; Nemotron-4 15B → 8B/4B at up301  to 40× fewer tokens than from-scratch; Llama-3.1-Minitron-4B-Depth distilled on ~380B tokens.302- **Assessment:** all of these *permanently* shrink the model — excellent baselines, but they303  answer the wrong question for us (they reduce total size, not the size↔residency coupling),304  and the strong ones require retraining budgets we don't have. Their *diagnostics* (Block305  Influence, activation-norm channel ranking) are directly reusable as residency scores.306307### 2.12 Sparsity + offload hybrids: M2Cache, Endor, RAP308309- **M2Cache (2410.14740):** neuron-level mixed precision + three-tier cache **HBM → DRAM → SSD**;310  important neurons fp16, colder ones more aggressively quantized, coldest on SSD; LRU at neuron311  granularity. The only published system with an explicit *SSD tier in a neuron cache hierarchy*.312  Server GPUs, not Mac.313- **Endor (2406.11674):** key observation for §4.3 — CSR-style formats for unstructured-pruned314  LLMs spend the saved bytes on indices, so offloaded pruned models are usually stored *dense*;315  proposes a hardware-friendly bitmap format so pruned weights actually transfer fewer bytes316  from SSD/flash. Directly applicable to any Mac NVMe streaming design.317- **RAP (2505.17138):** RL-guided *runtime* elastic pruning of weights + KV under a live memory318  budget — "pruning as a scheduling decision," another step toward pruning-as-policy.319320### 2.13 Quality reality check: Sirius321322- **Sirius (NeurIPS 2024):** systematic evaluation shows contextual-sparsity models hold up on323  prompt-understanding tasks but **significantly degrade on reasoning/deduction/knowledge324  (GSM8K, coding)**; yet sparse and dense models share problem-solving structure, and correcting325  only ~**11% of tokens** (dense-model verification, KV/hardware-efficient) restores full326  accuracy at ~78% of the theoretical efficiency gain.327- **Implication:** any localvm design that uses aggressive sparsity should budget a328  dense-verification / correction path (cheap on unified memory since CPU+GPU share the cache);329  perplexity alone will overstate quality (CATS/TEAL-style "99% retention" claims are mostly330  perplexity + short benchmarks, not multi-step reasoning).331332---333334## 3. Relevance to localvm-research335336### 3.1 How predictable are activation patterns, really? (measured numbers)337338- **Per-token sparsity (ReLU models):** 85% contextual sparsity (DejaVu, OPT); 2.4–3.1% of FFN339  neurons touched per token (LLM in a flash, OPT-6.7B/Falcon-7B); ~90% (ProSparse); <10% FFN340  activation in ReLU transformers generally (Lazy Neuron).341- **Temporal stability:** windowing over k=4–5 tokens works — after caching the last-4-token342  union, the per-token *delta* of new neurons is small enough to cut bytes/token from 6.7 GB343  (predictor alone) to **0.2 GB** (OPT-6.7B). This is a direct, measured confirmation of our344  Experiment B hypothesis on real hardware.345- **Spatial/structural stability:** power-law hot/cold split (PowerInfer); ~80% co-activation of346  Gate-Up-Down bundles per neuron (PowerInfer-2); strong cross-neuron co-activation exploitable347  by layout (Ripple); **flocking** — sequence-level shared expert sets (GRIFFIN, 50% FF params348  chosen once per prompt).349- **Predictor accuracy/cost:** low-rank per-layer predictors reach ~5% FN / 7% FP at <2.4%350  weight/FLOP overhead (LLM in a flash); but at scale predictors get heavy — 2.6 GB DRAM for a351  47B model (PowerInfer-2). Predictor-free thresholding (CATS/TEAL/DIP) eliminates this cost at352  some accuracy expense; ShadowLLM shows one early predictor can serve all layers (better353  accuracy and more prefetch lead time).354- **Open question for our Experiments A–C:** all the stability numbers above are from ReLU or355  ReLUfied models; nobody has published Jaccard/transition statistics for *thresholded SwiGLU*356  working sets (TEAL/CATS-style masks) — we should measure this ourselves on Llama-3/Qwen-class357  models before building anything.358359### 3.2 Does sparsity survive in non-ReLU models?360361Partially, and this is the field's central tension:362363- Exact zeros: essentially none in SwiGLU (DIP: "little inherent sparsity").364- *Approximate* sparsity: 40–50% model-wide prunable per token with minimal ppl loss (TEAL);365  50% FFN with ~99% task retention (CATS); 50% FF via flocking (GRIFFIN); and the recoverable366  ceiling with fine-tuning is ~85–90% (ProSparse/TurboSparse). Universal-properties (2509.00454)367  finds effective sparsity *increases with model size* — good news since our targets are big.368- But 40–50% ≠ 97%. For SSD paging, a 2× reduction in bytes/token is helpful yet far from the369  50× that ReLU windowing achieved. Bridging options: combine threshold sparsity with370  quantization (TEAL composes), low-rank hot path + sparse cold residual (R-Sparse suggests the371  decomposition), Sirius-style correction, or accept ReLUfied checkpoints where they exist.372- Quality caveat: Sirius's reasoning-degradation result means our quality metrics (§8.2) must373  include GSM8K/coding-style tasks, not just perplexity.374375### 3.3 What maps to Apple Silicon, and what doesn't376377| Assumption in prior work | Apple Silicon reality |378|---|---|379| GPU VRAM vs CPU DRAM split (DejaVu, PowerInfer) | No split — unified memory. The only meaningful hierarchy is **RAM vs NVMe** |380| PCIe transfer cost motivates hot/cold placement | Zero-copy CPU↔GPU; placement = *residency*, not device |381| 2:4 sparse tensor cores (Wanda/SparseGPT speedups) | **Absent on Apple GPUs** — N:M weight sparsity gives no free lunch; savings must come from avoided *loads* in custom Metal kernels |382| UFS 4.0: ~4 GB/s seq, ~1 GB/s 4K random (PowerInfer-2) | Apple NVMe: >6 GiB/s seq measured (M1 Max); ~2.25 GiB/s effective sparse reads at 32 KiB×32 threads — **the storage is 2–6× better than what PowerInfer-2 was built for** |383| Linux O_DIRECT/io_uring | macOS F_NOCACHE + many-threaded pread; APFS page cache behaves differently (our Experiment H) |384| Triton/CUDA gather kernels (TEAL, CATS, DejaVu) | **No Metal equivalents exist anywhere** (PowerInfer's Metal sparse backend: never shipped) |385| Rooted Android, offline per-device planner (PowerInfer-2) | Full user control of macOS; can pin memory (wired), use mmap, run calibration at "compile" time |386387### 3.4 The §4.3 key question: pruning as a cache policy — prior-art verdict388389Searched explicitly for "discarded weights live on SSD and are recovered on demand":390391- **Partially done:** LLM in a flash (sparsity-driven demand paging of FFN weights from flash —392  but ReLU-only, ≤2× oversubscription, no code, attention pinned); PowerInfer-2 + Ripple (same393  idea on Android/UFS with cluster caches and layout optimization); M2Cache (neuron LRU over394  HBM/DRAM/SSD with mixed precision); DIP (**mask chosen as a function of cache contents** —395  pruning literally becomes the cache policy, but only simulated mobile constraints);396  RAP (runtime pruning under memory budget, no SSD recovery); On-Demand Multi-Task Sparsity397  (2511.19986, edge, task-level sparse deltas from flash); VLM in a flash (2511.18692, neuron398  chunking for I/O-efficient VLM sparsification — shows the line is alive in 2025–26).399- **Not done anywhere (verified gap):** (a) using a **one-shot pruning importance score400  (Wanda/SparseGPT-style) as the *static tier-assignment* policy** — "pruned" weights demoted to401  NVMe in an Endor-style dense-readable format and *re-materialized* when a cheap online402  statistic (input norms, gate outputs, flocking profile) says they matter, restoring the dense403  model's quality ceiling instead of accepting permanent 7B-scale pruning damage;404  (b) any of this on **macOS / Metal / unified memory / Apple NVMe**; (c) combination with405  **layer-level** granularity (ShortGPT-scored cold layers paged in only when a router detects406  they're needed); (d) 4-bit-quantized neuron bundles on flash (PowerInfer-2 does 4-bit on UFS;407  LLM in a flash used fp16 — nobody did quantized bundles on Apple NVMe with Metal decode).408409### 3.5 What's unexplored (candidate experiment seeds)4104111. **Rebuild the LLM-in-a-flash measurement stack on our M5 Max** (Experiment H + E): reproduce412   the 32 KiB×32-thread flash-read curve, then measure TEAL-style 50% SwiGLU masks as a *paging*413   policy (not a FLOP policy) on Llama-3-8B/Qwen-14B class models. Nobody has published414   bytes/token for thresholded-SwiGLU paging.4152. **Working-set statistics for SwiGLU masks** (Experiments A–C): Jaccard(t, t+1), window-union416   growth curves, cross-prompt domain overlap — the numbers exist only for ReLU models.4173. **Wanda-score residency tiers + Sirius-style correction**: keep top-p% weights (by418   |W|·‖x‖ calibration) resident, stream the rest on demand from NVMe, dense-verify419   occasionally. This composes three verified results into a system nobody has built, on420   hardware (fast NVMe + unified memory + Metal) that is *more* favorable than anything prior421   work targeted.422423---424425## Sources426427- Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time — https://arxiv.org/abs/2310.17157 (accessed 2026-08-11)428- Deja Vu (OpenReview, ICML 2023) — https://openreview.net/forum?id=wIPIhHd00i (accessed 2026-08-11)429- PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU — https://arxiv.org/abs/2312.12456 (accessed 2026-08-11)430- PowerInfer (SOSP 2024 paper PDF, IPADS/SJTU) — https://ipads.se.sjtu.edu.cn/_media/publications/song-sosp24.pdf (accessed 2026-08-11)431- PowerInfer GitHub (macOS/Metal support status, supported ReLU models) — https://github.com/SJTU-IPADS/PowerInfer (accessed 2026-08-11)432- PowerInfer-2: Fast Large Language Model Inference on a Smartphone — https://arxiv.org/abs/2406.06282 (accessed 2026-08-11)433- PowerInfer-2 project page — https://powerinfer.ai/v2/ (accessed 2026-08-11)434- LLM in a flash: Efficient Large Language Model Inference with Limited Memory (Apple) — https://arxiv.org/abs/2312.11514 (accessed 2026-08-11)435- ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models (Apple, ICLR 2024) — https://arxiv.org/abs/2310.04564 (accessed 2026-08-11)436- ReLU Strikes Back — Apple Machine Learning Research page — https://machinelearning.apple.com/research/relu (accessed 2026-08-11)437- The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers — https://arxiv.org/abs/2210.06313 (accessed 2026-08-11)438- TEAL: Training-Free Activation Sparsity in Large Language Models — https://arxiv.org/abs/2408.14690 (accessed 2026-08-11)439- TEAL — Together AI blog — https://www.together.ai/blog/teal-training-free-activation-sparsity-in-large-language-models (accessed 2026-08-11)440- CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models (COLM 2024) — https://arxiv.org/abs/2404.08763 (accessed 2026-08-11)441- CATS GitHub — https://github.com/ScalingIntelligence/CATS (accessed 2026-08-11)442- GRIFFIN: Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generation (ICML 2024) — https://arxiv.org/abs/2404.01365 (accessed 2026-08-11)443- GRIFFIN GitHub — https://github.com/hdong920/GRIFFIN (accessed 2026-08-11)444- ShadowLLM: Predictor-based Contextual Sparsity for Large Language Models (EMNLP 2024) — https://arxiv.org/abs/2406.16635 (accessed 2026-08-11)445- ShadowLLM — ACL Anthology — https://aclanthology.org/2024.emnlp-main.1068/ (accessed 2026-08-11)446- ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models — https://arxiv.org/abs/2402.13516 (accessed 2026-08-11)447- Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters — https://arxiv.org/abs/2406.05955 (accessed 2026-08-11)448- Q-Sparse: All Large Language Models can be Fully Sparsely-Activated (NeurIPS 2024) — https://arxiv.org/abs/2407.10969 (accessed 2026-08-11)449- Sirius: Contextual Sparsity with Correction for Efficient LLMs (NeurIPS 2024) — https://arxiv.org/abs/2409.03856 (accessed 2026-08-11)450- SparQ Attention: Bandwidth-Efficient LLM Inference (ICML 2024) — https://arxiv.org/abs/2312.04985 (accessed 2026-08-11)451- SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot (ICML 2023) — https://arxiv.org/abs/2301.00774 (accessed 2026-08-11)452- Wanda: A Simple and Effective Pruning Approach for Large Language Models (ICLR 2024) — https://arxiv.org/abs/2306.11695 (accessed 2026-08-11)453- ShortGPT: Layers in Large Language Models are More Redundant Than You Expect — https://arxiv.org/abs/2403.03853 (accessed 2026-08-11)454- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024) — https://arxiv.org/abs/2310.06694 (accessed 2026-08-11)455- Compact Language Models via Pruning and Knowledge Distillation (Minitron, NVIDIA) — https://arxiv.org/abs/2407.14679 (accessed 2026-08-11)456- LLM Pruning and Distillation in Practice: The Minitron Approach — https://arxiv.org/pdf/2408.11796 (accessed 2026-08-11)457- SliceGPT: Compress Large Language Models by Deleting Rows and Columns (ICLR 2024) — https://arxiv.org/abs/2401.15024 (accessed 2026-08-11)458- LLM-Pruner: On the Structural Pruning of Large Language Models (NeurIPS 2023) — https://arxiv.org/abs/2305.11627 (accessed 2026-08-11)459- M2Cache: Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Caching — https://arxiv.org/abs/2410.14740 (accessed 2026-08-11)460- Ripple/Neuralink: Accelerating LLM Inference on Smartphones with Correlation-Aware Neuron Management / Neuron Co-Activation Linking — https://arxiv.org/abs/2410.19274 (accessed 2026-08-11)461- DIP: Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Masking (Qualcomm AI Research) — https://arxiv.org/abs/2412.01380 (accessed 2026-08-11)462- Endor: Hardware-Friendly Sparse Format for Offloaded LLM Inference — https://arxiv.org/pdf/2406.11674 (accessed 2026-08-11)463- SparseInfer: Training-free Prediction of Activation Sparsity for Fast LLM Inference — https://arxiv.org/pdf/2411.12692 (accessed 2026-08-11)464- Post-Training Statistical Calibration for Higher Activation Sparsity — https://arxiv.org/pdf/2412.07174 (accessed 2026-08-11)465- R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference — https://arxiv.org/abs/2504.19449 (accessed 2026-08-11)466- Spark Transformer: Reactivating Sparsity in FFN and Attention (NeurIPS 2025) — https://arxiv.org/html/2506.06644v2 (accessed 2026-08-11)467- Universal Properties of Activation Sparsity in Modern Large Language Models — https://arxiv.org/abs/2509.00454 (accessed 2026-08-11)468- RAP: Runtime Adaptive Pruning for LLM Inference — https://arxiv.org/pdf/2505.17138 (accessed 2026-08-11)469- DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs — https://arxiv.org/html/2506.20194 (accessed 2026-08-11)470- Motivating Next-Gen Accelerators with Flexible (N:M) Activation Sparsity — https://arxiv.org/pdf/2509.22166 (accessed 2026-08-11)471- VLM in a flash: I/O-Efficient Sparsification of Vision-Language Model via Neuron Chunking — https://arxiv.org/html/2511.18692 (accessed 2026-08-11)472- On-Demand Multi-Task Sparsity for Efficient Large-Model Deployment on Edge Devices — https://arxiv.org/pdf/2511.19986 (accessed 2026-08-11)473- Fast Forward: Accelerating LLM Prefill with Predictive FFN Sparsity — https://arxiv.org/pdf/2602.00397 (accessed 2026-08-11)474- Dynamic sparsity in tree-structured feed-forward layers at scale — https://arxiv.org/pdf/2604.08565 (accessed 2026-08-11)475