# project: localvm-research document: research/state_of_the_art author: Simon-Pierre Boucher contact: contact@spboucher.ai created: 2026-08-12 status: draft

# State of the Art: Post-Training Execution of Over-Budget LLMs on Apple Silicon

Phase 2 deliverable (charter §5). Synthesizes the five Phase 1 notes (research/notes/{quantization, sparsity_pruning, out_of_core_memory_systems, decomposition_progressive, speculation_error_stability}.md). Every source cited here is listed with URL and access date in research/bibliography.md. Per-technique blocks use the mandated fields; abbreviations: Mod = model modification required, Retrain = retraining required, Mem/BW/Compute = memory / bandwidth / compute reduction, AS/Metal = works on Apple Silicon (Metal), OSS/macOS = open-source implementation and whether it builds on macOS arm64.

Organizing frame (charter §2): total model size ≠ resident size ≠ bytes read per token ≠ parameters materially required for this token. Each taxonomy family below decouples a different pair of these quantities — or fails to.

# 0. Taxonomy

# 1. Substrate facts: Apple Silicon target (M5 Max, 48 GB, AP2048Z NVMe)

Measured and documented constraints that gate every technique below.

• Unified memory bandwidth: ~460–614 GB/s on M5 Max (Apple specs, config-dependent). CPU and GPU share it — no VRAM/DRAM split, no PCIe. Bandwidth-bound decode ceiling ≈ memory BW ÷ resident-active bytes (fully resident Q4-70B ≈ 40 GB → ~11 tok/s ceiling).
• Internal NVMe (measured, expH, this repo, 2026-08-12): ~13 GB/s peak random read at 1 MiB blocks QD8; 67 MB/s at 4 KiB QD1. That is ~35:1 RAM:SSD at best and ~7000:1 worst-case — small random reads are the cliff, not sequential bandwidth. Our peak exceeds the literature's Apple-NVMe priors (LLM in a flash: >6 GiB/s sequential, ~2.25 GiB/s effective sparse on M1 Max), so paging economics are more favorable on our hardware.
• Concurrent Metal GPU matmul load reduces SSD random-read throughput by <5% (expH, this repo, 2026-08-12) — compute/I-O overlap is essentially free; memory-controller contention is a non-issue.
• F_NOCACHE does not bypass already-resident pages (expH discovery, this repo, 2026-08-12; consistent with Apple dev forums #25464 and fio #48). There is no O_DIRECT, no posix_fadvise, no io_uring; eviction from the Unified Buffer Cache is opaque approximate-LRU. Cold-state experiments require purge(8) + vm_stat deltas.
• Page size is 16 KB — the minimum paging/fault unit; safetensors offsets are not page-aligned (root cause of MLX's mmap dead-end), GGUF's are. makeBuffer(bytesNoCopy:) requires page alignment → any compiled weight format must 16 KB-align every independently pageable block.
• Zero-copy path exists and ships: llama.cpp's Metal backend reads mmap'd file pages directly via MTLResourceStorageModeShared (discussion #21223) — SSD→page-cache→GPU with no copy, impossible on discrete GPUs. Purgeable MTLHeap + residency sets give a three-tier RAM hierarchy (wired-hot / purgeable-warm / evictable-cache) unique to macOS.
• Wired-memory cliff: iogpu.wired_limit_mb defaults to ~66–75% of RAM; unbounded wiring ends in IOGPUMemory kernel panic (mlx-lm #883), not graceful degradation.
• macOS compressor is useless on weights (quantized weights ≈ incompressible entropy); weight overflow must be file-backed (dropped), never anonymous (compressed/swapped).
• Decode-kernel cost rule (i-quant lesson, llama.cpp #5617): on Apple GPUs, fewer bits → faster tokens only if decode stays shift/mask-cheap. IQ-quants read ~half the bytes of Q4_0 yet run slower on M2 Max (53.9 vs 63.1 tok/s); LUT-heavy decode is compute-bound on Metal. Bitplanes and bitshift-trellises qualify; big codebook LUTs do not.

# 2. Static compression

Every technique here shrinks the checkpoint uniformly; all bytes of the compressed model are still read every token — memory and bandwidth reductions are proportional and fixed at encode time.

2.1 Scalar weight-only PTQ — GPTQ, AWQ, HQQ, SqueezeLLM

• Problem: checkpoint capacity + proportional decode bandwidth. Mod: re-encoded weights only. Retrain: none (calibration; HQQ calibration-free, 70B in <5 min).
• Mem: ~4× @4-bit, ~5.3× @3-bit. BW: proportional (GPTQ: 3.25–4.5× generation speedup vs FP16). Compute: neutral (fused dequant). Latency: ∝ bytes.
• Quality: near-lossless @4-bit (GPTQ OPT-1.3B: FP16 PPL 14.63 → 15.47; naive RTN 48.24). SqueezeLLM 3-bit LLaMA-7B: 18.08 → 7.75 via sensitivity-weighted k-means + 0.05–0.45% FP16 sparse outliers. AWQ more calibration-shift-robust than GPTQ (0.5–0.6 vs 2.3–4.9 PPL under shift).
• AS/Metal: algorithms platform-neutral; outputs convert to GGUF/MLX. Fast kernels (Marlin/ExLlama/SqueezeLLM) CUDA-only. OSS/macOS: mlx-lm ships GPTQ/AWQ-style recipes natively (yes).
• Limitation: uniform scalar grids wall below 3 bits. Extension: GPTQ Hessians and SqueezeLLM's dense+sparse split are reusable for allocating residual precision.

2.2 GGUF block formats (llama.cpp k-quants / i-quants / MXFP4)

• Problem: same, plus per-tensor precision mixing. Mod: convert-time. Retrain: none (i-quants need imatrix calibration).
• Mem/BW: proportional (Q4_K_M ≈ 4.9 GB for 8B, ~+0.08 PPL). Quality/bit: k-quants beat MLX affine at matched bpw — Q4_K_M 4.88 bpw = 0.0208 nats KL-to-FP16 vs MLX affine 4-bit 0.0577 (Feldman 2026, ~2.8×).
• AS/Metal: all types have hand-written Metal kernels, but i-quants pay the LUT penalty (§1). OSS/macOS: yes, first-class.
• Limitation: static per-tensor decisions; no partial loading. Extension: imatrix = ready-made per-block sensitivity signal.

2.3 MLX native quantization + learned quants (affine 2–8 bit; DWQ, dynamic_quant)

• Problem: Apple-first quantized inference. Mod: convert-time. Retrain: none (DWQ distills quant params only; ≈ +0.6 effective bits).
• Mem/BW: proportional; MLX 4-bit 8B ≈ 4.5 GB. Latency: within ±10–20% of llama.cpp; MLX wins 4-bit decode, llama.cpp prefill.
• Quality: affine 4-bit/g64 trails Q4_K_M; g32/DWQ closes gap. AS/Metal: fully native, fused dequant-matmul, quantized-KV. OSS/macOS: yes — the maintained Apple-first PTQ toolchain, pure Python, hackable.
• Limitation: uniform affine only; no codebooks, no sub-2-bit, no nested layout; precision fixed at convert time. Extension: the mode plug-point + custom Metal kernels is where a progressive format would live; DWQ loop is the vehicle for residual-aware calibration.

2.4 Rotation-based W+A quantization — SmoothQuant, QuaRot, SpinQuant

• Problem: activation outliers blocking W4A4/W8A8. Mod: invariance rotations absorbed offline (output-preserving). Retrain: none (SpinQuant learns rotations, no model retrain).
• Mem: checkpoint unchanged (weights-side same as 2.1); KV shrinks. BW: mainly KV + prefill. Compute: INT4/8 matmul density — a prefill/throughput play, mostly irrelevant to batch-1 decode BW on Mac.
• Quality: QuaRot Llama-2-70B W4A4 ≤0.47 PPL loss, 99% zero-shot retention. AS/Metal: no shipped Metal W4A4 path; MLX quantize_input is the beginning. OSS/macOS: reference code CUDA.
• Limitation: wrong bottleneck for us. Extension: incoherence preprocessing is upstream-compatible with any base encoding (QuIP#/QTIP depend on it).

2.5 Codebook / lattice / trellis quantization — QuIP#, AQLM, GPTVQ, QTIP, EXL3, NestQuant

• Problem: the 2–2.5 bpw frontier. Mod: heavy re-encoding. Retrain: calibration hours; quality depends on fine-tuning (AQLM ~720 GPU-h for 70B).
• Mem: ~8× (2 bpw usable: QuIP# 70B 2-bit Wiki2 PPL 4.16 vs FP16 3.12; EXL3 70B coherent at 1.6 bpw in <16 GB). BW: conditional on decode cost — QTIP >80% of peak GPU BW (bitshift trellis, ~2 instr/weight); AQLM's 1 MiB codebooks blow L1 (20.6 vs 106.3 tok/s QTIP-class). GPTVQ is the one VQ validated on unified-memory Arm CPU with simultaneous DRAM+latency wins.
• Quality: SOTA per bit below 3; first regime where 3-bit scales better than 4-bit (QuIP#). AS/Metal: no Metal implementation of any of these exists; i-quant evidence predicts LUT variants lose on Apple GPUs. OSS/macOS: CUDA-only.
• Limitation: fixed bitrate at encode; CUDA ecosystem. Extension: RVQ/additive structure is inherently progressive (→ §5); a LUT-free Metal trellis decoder is unclaimed.

2.6 Extreme low-bit — BiLLM, BitNet/bitnet.cpp, ParetoQ, EfficientQAT

• Problem: ≤2-bit regimes. Mod/Retrain: BiLLM PTQ (none); BitNet trained-from-scratch (excluded by charter §14); ParetoQ/EfficientQAT = QAT finetuning.
• Mem: 8–16×. Quality: BiLLM ~1.08 bpw 70B PPL 8.41 (clearly degraded); EfficientQAT w2g64 7B PPL 6.86 vs 5.47 FP16, 70B −3% acc in 41 A100-h — the realistic "2-bit that still behaves like the original" ceiling. ParetoQ: sharp representational transition between 2 and 3 bits — ≤2-bit leaves the pretrained basin.
• AS/Metal: bitnet.cpp is CPU-only (NEON LUTs) yet runs a 100B ternary model at 7.45 tok/s on M2 Ultra — existence proof that sub-2-bit makes 100B-class Mac-feasible. OSS/macOS: bitnet.cpp yes; others CUDA/PyTorch.
• Limitation: pure-PTQ sub-2-bit destroys quality; QAT violates our budget. Extension: fixes the design floor — a base representation must be ≥2-bit-effective (safer 2.5–3) to preserve routing/decision structure without retraining.

2.7 KV-cache quantization — KIVI, KVQuant, Coupled Quantization

• Problem: KV bytes/token rival weight bytes at long context. Mod: runtime cache format. Retrain: none.
• Mem/BW: KIVI 2-bit (K per-channel, V per-token) 2.6× peak-mem; KVQuant 3-bit <0.1 PPL, 1M-token context on one A100; CQ ~1 bit/channel-equiv.
• AS/Metal: the solved part of the working-set problem on Macs — llama.cpp --cache-type-k/v with flash attention on Metal; mlx-lm --kv-bits {2,4,8}; TurboQuant Metal kernels at 3.25/4.25 bits landing. OSS/macOS: yes.
• Limitation/Extension: any weight working-set argument must co-model KV; asymmetric K/V precision (KVSplit) already pays on Metal.

2.8 SVD family & structural decomposition — SVD-LLM, SliceGPT, LASER

• Problem: low-rank re-representation. Mod: factorized/sliced weights. Retrain: none (calibration/whitening).
• Mem/BW: parameter-ratio-proportional, dense (plain GEMMs). Quality: weights are not globally low-rank — SVD-LLM 7.73 PPL vs ASVD 11.14 at 20% compression (LLaMA-7B), but ≥40% costs large downstream accuracy. SliceGPT: −25% params at 99% zero-shot (Llama-2-70B), up to 1.55× throughput — the only pruning-family transform that trivially runs fast on Metal (smaller dense GEMMs). LASER: replacing selected late-MLP matrices with low-rank improves accuracy up to +30 pts — high-order components of specific matrices are noise-like.
• AS/Metal: excellent (dense matmul; MLX-native, composable with affine quant). OSS/macOS: yes (PyTorch, ports trivial).
• Limitation: fixed rank at compile; residual discarded despite carrying most spectral energy. Extension: low-rank resident hot path + SSD-resident residual — none of these papers stores the residual at all.

2.9 Low-rank + quantized residual — CALDERA

• Problem: 2–2.5 bpw via W ≈ Q + LR with error bounds. Mod: re-encode. Retrain: calibration-aware alternating minimization.
• Mem: 2–2.5 bpw class. Quality: competitive at extreme ratios. AS/Metal: LR is dense (fine); Q uses QuIP#-style lattices (CUDA). OSS/macOS: no.
• Limitation: single operating point; joint optimization couples terms — dropping one is not quality-graceful. Extension: re-derive with a progressivity constraint (base alone usable) — the bridge to §5.

2.10 Cross-layer sharing & delta compression — Basis Sharing, DeltaLLM, Relaxed Recursive Transformers, BitDelta, Delta-CoMe

• Problem: redundancy between matrices. Mod: shared trunk + per-layer corrections. Retrain: light (DeltaLLM ~30–40M tokens; RRT distillation; BitDelta minutes).
• Mem: modest as compression (12–25%); as evidence enormous: finetune deltas quantize to ~1 bit/param near-losslessly (BitDelta; Delta-CoMe holds on math/code) — proof that "model = reference + extremely compressible correction" is real structure. BitDelta iterated on itself yields a monotone stack of 1-bit masks → an accidental progressive code.
• AS/Metal: sign-mask + scale decode is trivially Metal-friendly; Basis Sharing/DeltaLLM are dense ops. OSS/macOS: PyTorch, portable.
• Limitation: zero-shot layer tying degrades; ratios far from 10×. Extension: shared trunk as cache-resident core, deltas as SSD pages — untried as a memory-hierarchy assignment.

2.11 Lossless & generative recompression — DFloat11, SeedLM

• DFloat11 (lossless Huffman of BF16 exponents): 30% size cut, bit-identical, but 2–3× slower tokens/s than uncompressed-in-VRAM — lossless coding shrinks capacity, not effective bandwidth; entropy decode sits on the critical path. CUDA-only. The cautionary datapoint for any decode-on-load design.
• SeedLM (Apple): weights → LFSR seed + coefficients; regenerate at inference — trades bandwidth for free compute, exactly the right trade on BW-bound hardware (~4× FPGA at 70B, data-free, 4-bit parity). No Metal port. Extension: "weights need not be stored, only recoverable" — Apple-authored, conceptually adjacent to our thesis.

2.12 One-shot & structured pruning — SparseGPT, Wanda, ShortGPT, Sheared-LLaMA, Minitron

• Problem: permanent weight removal. Mod: destructive. Retrain: none (one-shot) to heavy (Sheared/Minitron).
• Quality: 50% unstructured nearly free only at ≥65B (LLaMA-65B 3.56→4.57; 7B 5.68→7.26, ~27% worse); 2:4 much worse at small scale (7B → 11.02). ShortGPT: middle-late layers highly redundant; "Secretly Linear": Procrustes linearity ≈0.99 between consecutive layers.
• AS/Metal: Apple GPUs have no sparse tensor cores — unstructured sparsity yields zero BW savings unless the representation skips loads (CSR indexing overhead cancels it — Endor's point). OSS/macOS: algorithms portable; speedups are not.
• Limitation: answers the wrong question (total size, not residency), destroys quality at 7B scale. Extension: importance scores as residency policy — Wanda's |W|·‖x‖ is cheap enough to rank weights for RAM-vs-NVMe instead of deletion; Block Influence for layer-level tiers; Endor's bitmap format for dense-readable demoted weights.

# 3. Sparsity & conditional execution

These decouple parameters required per token from total parameters — the precondition for any paging win (§4).

3.1 Contextual sparsity — DejaVu, ShadowLLM

• Problem: which heads/neurons does this token need. Mod: none to the model; per-layer MLP predictors added. Retrain: predictors only.
• Mem: none (model stays resident). BW: up to 85% contextual sparsity ⇒ proportional HBM traffic cut; >2× vs FasterTransformer on OPT-175B, no measured quality loss. ShadowLLM: one early-layer predictor shadows all layers, >15% accuracy over DejaVu criteria at equal sparsity, +20% speed — and gives maximum prefetch lead time.
• AS/Metal: ReLU-era OPT models; multi-A100 assumptions; no Metal kernels. OSS/macOS: CUDA.
• Limitation: ReLU-dependent; model must fit in accelerator memory. Extension: "slowly changing hidden states" as an NVMe prefetch signal rather than a FLOP-skip signal.

3.2 ReLUfication — ReLU Strikes Back, ProSparse, TurboSparse, Q-Sparse

• Problem: SwiGLU killed exact zeros. Mod: activation swap. Retrain: yes (fine-tuning to ~150B tokens) — violates post-training-only, but checkpoints (ProSparse-LLaMA, TurboSparse-Mistral/Mixtral, Bamboo) are downloadable test vehicles.
• Numbers: 87.9–89.3% sparsity at parity (ProSparse); TurboSparse-Mistral-7B activates 2.5B params/token, Mixtral-47B activates 4.3B. Sets the recoverable ceiling (~85–90%).

3.3 Training-free SwiGLU sparsity — CATS, TEAL, GRIFFIN, DIP

• Problem: sparsity on stock modern models, no retraining. Mod: none. Retrain: none (calibration thresholds).
• BW: CATS 50% FFN sparsity at ~99% task retention (≈25% model-wide); TEAL 40–50% model-wide (all matrices) with minimal degradation, 1.53–1.8× decode via Triton; GRIFFIN "flocking" — 50% of FF params selected once per sequence (prefetch-friendly granularity); DIP: cache-aware masks — sparsity chosen given DRAM contents: 46% less memory, 40% more throughput, <0.1 PPL on Phi-3-Medium under mobile limits — the first explicit "sparsity-as-cache-policy" formulation.
• Quality caveat (Sirius, NeurIPS 2024): contextual sparsity degrades reasoning/GSM8K/coding specifically; correcting ~11% of tokens by dense verification restores accuracy at ~78% of the efficiency gain. Perplexity-based "99% retention" claims overstate quality.
• AS/Metal: all kernels CUDA (Triton); no Metal equivalents exist anywhere. OSS/macOS: no.
• Limitation: 40–50% ≠ 97% — a 2× bytes/token cut, not 50×. Extension: TEAL as the first Metal port target; thresholded-SwiGLU masks as paging policy — bytes/token for this has never been published; working-set stats (Jaccard, reuse distance) exist only for ReLU models → our Experiments A–C.

3.4 SparQ Attention (attention-side analogue)

• Fetch only KV rows whose keys matter for the current query: up to 8× attention data-transfer cut, negligible loss, no fine-tuning. Demand-driven fetching works for state, not just weights; complements §2.7.

# 4. Out-of-core & paging systems

4.1 Dense streaming — FlexGen, ZeRO-Inference, AirLLM

• Problem: capacity only. Mod/Retrain: none. Mem: resident ≈ 0 possible. BW: bytes/token = full checkpoint unless batched — the hard ceiling: tokens/s ≤ storage BW ÷ non-resident bytes. On our SSD a 70 GB non-resident model can never exceed ~0.1–0.2 tok/s dense.
• Latency: FlexGen OPT-175B on 16 GB GPU ≈ 1 tok/s effective only at huge batch; AirLLM 70B in 4 GB, 15–30 min/response. AS/Metal: CUDA (FlexGen LP idea portable). Verdict: proof that capacity decoupling alone is worthless interactively; these are the harness straw-man controls.

4.2 LLM in a flash (Apple, ACL 2024) — closest prior art, ran on our platform

• Mod: predictors added (<2.4% overhead); ReLU-family models only. Retrain: predictors only.
• Mem: DRAM ≈ 52% of model (attention+embeddings pinned; FFN on flash). BW: bytes/token 13.4 GB naive → 0.2 GB with predictor+windowing (OPT-6.7B, k=4 window; 2.4–3.1% of FFN neurons touched/token) — the single most important measured number in the literature for charter §8.3.
• Latency: I/O per token 2196 ms → 105 ms on M1 Max; 4.23× Metal/M1, 7.44× Metal/M2 Ultra; runs models ~2× DRAM. Effective flash throughput ~2.25 GiB/s with row-column bundling at ≥32 KiB × 32 threads.
• AS/Metal: yes — measured on M1 Max / M2 Ultra with Metal. OSS/macOS: no code released; nothing in MLX/llama.cpp implements it.
• Limitation: ReLU-only (SwiGLU "not addressed"); attention never paged; ≤2× oversubscription; fp16 bundles (no quantization co-design); negative result: co-activation bundling failed (hot neurons re-loaded — later solved by Ripple's global placement).
• Extension: reproduce its throughput surface on our AP2048Z (expH); redo for SwiGLU/MoE with 4-bit bundles and Metal compute.

4.3 PowerInfer / PowerInfer-2 / Ripple

• PowerInfer (SOSP 2024): power-law hot/cold neuron split, hot→VRAM, cold→CPU; 13.2 tok/s OPT-175B-class on one 4090. GPU/CPU split is meaningless on unified memory; Mac analogue is RAM/NVMe. macOS status verified: CPU-only, "limited" gains; Metal sparse backend planned, never shipped.
• PowerInfer-2: 47B (TurboSparse-Mixtral) at 11.68 tok/s on a 24 GB phone over UFS 4.0 (~1 GB/s random) — the closest architectural template for a Mac (single memory pool + flash). Design lessons: segmented neuron cache; Gate-first two-phase reads; predictors cost 2.6 GB DRAM at 47B; requires ReLUfied models; P99 +40.9%. Our NVMe is ~4–13× faster than its storage — economics transfer favorably; nobody has built it.
• Ripple: co-activation-aware flash layout (offline global placement) — belongs in our compile stage.
• OSS/macOS: PowerInfer yes-but-CPU-only; PowerInfer-2 closed, Android.

4.4 M2Cache

• Neuron-level mixed precision × tier unified: hot neurons FP16 in HBM, colder quantized in DRAM, coldest on SSD; up to 14× vs baseline offload, 70B on 24 GB VRAM. The closest published architecture to charter §13 — but Linux/CUDA, PCIe assumptions, static importance ranking, and no refinement: a neuron is fetched at one precision, never upgraded. OSS/macOS: no.

4.5 llama.cpp mmap + Metal (the working macOS evidence)

• Mod/Retrain: none. Mechanism: GGUF mmap'd; Metal reads file pages zero-copy; MoE overflow works by OS demand paging.
• Measured (discussion #18758, M5 Pro/AP1024Z, Qwen3-Next-80B): replacing fault-streaming with layout-aware explicit slice reads: 1418 → 370 reads/token, 2.23× cold-decode I/O, +13–14% end-to-end; mmap beat direct I/O because the page cache retains the hot expert working set.
• Limitation: replacement policy is the kernel's (opaque LRU, scan-vulnerable); no model-aware prefetch even though the router decision is known before the expert runs; two-tier expert cache is an open, unfilled feature request (#20757). OSS/macOS: yes, first-class.

4.6 MLX status

• Lazy loading only; no paging mechanism at all. Community mmap prototype: 70 GB model on 64 GB → 0.025 tok/s (kernel LRU vs cyclic dense scans = pathological worst case). This negative result defines our problem. Wired-limit/residency-set controls exist; everything else must be built by us.

4.7 MoE offloading — Eliseev & Mazur, cache-conditional experts, MoBiLE, SolidAttention

• Eliseev–Mazur: LRU expert cache + speculative expert prefetch (apply layer k+1's gate to layer k's hidden state) → Mixtral-8x7B at 2–3 tok/s on 11–16 GB GPUs. "Branch prediction for weights" in embryo; reuse/prefetchability confirmed general (arXiv 2511.05814).
• Cache-conditional experts: router biased toward resident experts — the model adapts to the memory system. MoBiLE: misses served by smaller substitute experts — never block compute. SolidAttention (FAST 2026): KV on SSD; paging unit chosen by the storage medium, not model granularity (3.1×, 98% KV memory cut at 128k).
• OS/DB imports with no LLM instantiation (from notes §4): MRU for cyclic weight scans (DBMIN 1985 — LRU is worst-case for dense decode, MRU is the theoretical floor; apparently never applied); ARC/ghost-list expert caches; per-tensor-class buffer pools; anti-caching's "never block on a miss"; Pythia-style learned prefetch; working-set admission control (Denning); purgeable-MTLHeap kernel-cooperative caches; five-minute-rule residency economics.

# 5. Progressive / nested representations ⭐

The family that directly attacks bytes stored ≠ bytes read. All of it is 2024–2026.

5.1 Any-Precision LLM (ICML 2024 oral) / Matryoshka Quantization (ICLR 2025 oral)

• Problem: one artifact, many precisions. Mod: bitplane / MSB-nested re-encoding. Retrain: Any-Precision none (incremental upscaling from a 3-bit seed, <1 min for 7B); MatQuant QAT or OmniQuant-style PTQ.
• Mem: {3..8}-bit Llama-2-7B in 8.4 GB vs 29.9 GB separate (3.56×). BW: bytes/token = f(precision requested), decoupled from storage — "reduced bit-width directly translates into proportional speedup, as we simply load the specified number of bits." Quality: each slice matches dedicated models; MatQuant int2 slice up to +10% over dedicated int2 QAT; int2-FFN Gemma-2 9B beats int8-FFN Gemma-2 2B.
• AS/Metal: none — engines are CUDA (bitplane layout, bit-transpose kernels); bitplane decode is bit-manipulation-heavy, Apple-GPU cost unknown (→ Experiment D/E). OSS/macOS: no.
• Limitation: precision scheduled globally and statically; full chosen precision loaded for every weight every token; all planes kept resident.

5.2 BitStack (ICLR 2025) / RRQ / Drop-by-Drop

• BitStack: training-free iterative significance-weighted decomposition → ~1-bit residual blocks, universally sorted by importance; runtime loads as many as memory allows → megabyte-granular quality↔residency tradeoff, matching/beating GPTQ/AWQ at extreme ratios. Known weakness: on-the-fly reconstruction slows inference — reconstruction must be fused into the matmul.
• RRQ (2026): calibration-free 2-bit base + stacked 2-bit RTN residuals; multi-precision package for Qwen3-8B in ~1293 s. Drop-by-Drop (2026): AQLM-style codebooks with Matryoshka supervision, grounded in successive-refinement theory; ordered codebooks droppable at inference; explicitly motivates "hardware-aware dynamic loading" — the closest published object to a progressive residual base representation. No systems implementation, no SSD tier, no Apple port.
• BitStack is an unwitting rediscovery of embedded wavelet coding (EZW/SPIHT/EBCOT); nobody has connected weight encodings to R-D-optimal per-block truncation, Nanite-style selective residency, or GMRES-IR-style residual correction (mature machinery in graphics/HPC/DB fields).

5.3 Phase-/token-adaptive precision — PMPD, QuickSilver, FlexQuant, DP-LLM, MoBiQuant

• PMPD (ICLR 2025): precision lowered as generation deepens, task/prompt-adaptive schedulers; 3.8–8.0× decode on an NPU — validates that required precision is position-dependent and runtime-schedulable. QuickSilver/FlexQuant/DP-LLM/MoBiQuant: entropy/sensitivity-gated per-token bit-width over Matryoshka overlays.
• All are feed-forward heuristics: nothing verifies the low-precision tokens afterward — lossy, empirical-only. And all keep every precision level resident. Confirms the mechanism is implementable; none closes the loop with exactness or with storage.

Common ledger for §5: no member selects precision per weight-block per token; none pages residuals from SSD; none reports bytes-read-per-token (they report resident size); none has Metal kernels.

# 6. Speculation, verification & decision stability

6.1 Classical + head-based speculative decoding — Leviathan/Chen, Medusa, EAGLE-3, ReDrafter

• Problem: amortize expensive-model reads over multiple tokens. Mod: none (draft model) or trained heads. Retrain: heads only.
• Exactness: rejection sampling provably preserves the target distribution — the canonical proof that approximate computation + cheap correction = exact computation. Numbers: 2–3× (Leviathan; greedy > sampling — argmax easier to reproduce); EAGLE-3 3.0–6.5×, acceptance length up to 7.5; production α ≈ 0.6–0.8.
• AS/Metal: verified on Apple hardware — ReDrafter on MLX: 1.37× M1 Max, 2.3× M2 Ultra; llama.cpp --model-draft and mlx_lm ship speculation on Metal. OSS/macOS: yes.
• Limitation: verification granularity = whole target model over a token block.

6.2 Depth self-speculation — Draft&Verify, LayerSkip, Kangaroo, SWIFT, CLaSp

• Draft = same weights, fewer layers; verify = all layers; lossless. 1.2–2.16×. Measured acceptance: 68.9% (Llama-2-7B drafting at layer 8/32), 74.5% (13B at 15/40); CLaSp: Llama-3-70B tolerates skipping 44/80 layers. Tuned Lens "prediction depth": many tokens stabilize well before the final layer. Retrain: LayerSkip yes; SWIFT/CLaSp none. AS/Metal: control flow, trivially portable. Limitation: the draft axis is depth, never precision-of-weights-with-storage-tier.

6.3 Precision self-speculation — QSpec, QuantSpec, ML-SpecQD

• Draft and target share the same weights at different precisions: QSpec W4A4 draft / W4A16 verify, 1.64×, no quality loss, no training; QuantSpec (Apple+Berkeley, ICML 2025): 4-bit hierarchical-KV+weights draft, full-precision verify, >90% acceptance, ~2.5×, edge-motivated; ML-SpecQD: MXFP4 cast as turnkey draft, >2×.
• This proves the core premise: a 4-bit cast of the same model agrees with the 16-bit model on the large majority of tokens, and disagreements are exactly repairable by shared-weight verification. What none do: exploit agreement to avoid storing/loading high-precision weights — both operand sets stay resident; the win is arithmetic, not capacity or bytes/token. OSS/macOS: CUDA.

6.4 Speculation × offloading — SpecInfer, SpecExec

• SpecInfer: 2.6–3.5× specifically for offloading-based inference. SpecExec: huge draft trees verified by an offloaded 70B in one pass — Llama-2-70B at 4–6 tok/s (4-bit) on consumer GPUs, 10.6–18.7× over sequential offloaded decoding, ~20 accepted tokens per full-model weight sweep. The key economics result: when reading the model costs seconds, speculation divides bytes/token by the acceptance length. OSS/macOS: CUDA, concepts portable.

6.5 Confidence-gated escalation (lossy) — BiLD, cascades, CALM

• BiLD: small model decodes; fallback/rollback to the large model on low confidence — 2.12×, no exactness guarantee. FrugalGPT/RouteLLM: ~1 in 6 queries needs the big model. CALM (NeurIPS 2022): per-token early exit with distribution-free risk calibration — sequence-level consistency with the full model provably maintained at chosen ε; ~3× decode. The strongest published answer to "can computation stop when more precision can't change the output" — answered statistically, for the depth axis only. Nobody has done CALM for weight precision / residual count.

6.6 Formal error bounds (§4.9 verdict: vacuous at scale)

• Self-attention is not globally Lipschitz (Kim et al. 2021); local constants grow ~√n in sequence length (Castin et al.) and compose to astronomically loose logit bounds over 30–80 layers. CROWN/auto_LiRPA supports weight perturbations in principle but is O(m²n³) — BERT-small scale; DeepT certifies a few layers in minutes. 4–5 orders of magnitude from a 7B decoder.
• Usable primitives anyway: the margin certificate (argmax stable iff top-2 margin > 2·‖Δlogits‖∞ bound) is trivial once any bound exists; QEP (NeurIPS 2025): layer-wise quantization errors compound near-exponentially with depth; EMNLP 2025: residual-stream magnitudes predict which inputs break quantization (ρ = 0.82) → a cheap learned per-input error predictor + CALM calibration is the realistic instrument. GMRES-IR (Carson–Higham): the canonical "factorize cheap, refine with residuals, guaranteed" pattern — never written down for transformer inference; probabilistic error analysis (√n·u constants) replaces vacuous worst cases.

6.7 Measured decision stability (the §4.10 evidence base)

• Same-top-token at 4-bit-class quantization: 90.87–91.23% (llama.cpp KLD tooling; mean KLD ≈ 0.054, 99th pct 0.53, max ~22–26 — heavy tail). MLX: mean KLD 0.014 (8-bit) → 0.059–0.113 (4-bit variants); MoE router protection changes rankings — which weights get precision matters more than average bpw.
• "Accuracy is Not All You Need" (Microsoft): models within 1% aggregate accuracy flip up to ~15% of individual answers (25%+ for layer-dropping); flips correlate with KL at Spearman 0.96–0.97 → aggregate benchmarks cannot certify a compressed runtime; KL/flips/agreement are the quality currency (charter §8.2).
• Fragility floor: batch-size-dependent rounding alone (~1e-5 relative) diverges greedy completions (Thinking Machines) — a persistent 5–15% of tokens sit on a knife's edge, and they are disproportionately the load-bearing ones. The joint distribution (cheap-model margin × agreement with full model) has never been published — Experiment G fills a real gap.

# 7. Overlap analysis

# 7.1 Combinations already tried (do not reinvent)

# 7.2 Seemingly novel ideas that are actually known

• "Treat weights like virtual memory" — llama.cpp mmap has done it since 2023, and the MLX prototype measured its failure mode (0.025 tok/s): known, including why naive versions die (kernel LRU × cyclic scans).
• "Hot/cold weight split by activation frequency" — PowerInfer's power-law statistics (SOSP 2024).
• "Escalate to the expensive model only when uncertain" — BiLD (2023), cascades, CALM (with calibrated guarantees, depth axis).
• "Draft with a cheap version of the same model, verify exactly" — QSpec/QuantSpec own "cheap numerics, exact verify"; LayerSkip owns "cheap depth, exact verify".
• "Choose the sparsity mask based on what's already cached" — DIP (Qualcomm, 2024).
• "Store base + refinable residuals of weights" — Any-Precision/MatQuant/BitStack/RRQ/Drop-by-Drop; also EZW/JPEG2000 thirty years ago.
• "Quality-vs-resident-size as a runtime knob" — BitStack demonstrates it (statically) end to end.
• Known negative results to respect: lossless decode on the critical path costs 2–3× (DFloat11); LUT-heavy decode loses on Metal (i-quants); naive co-activation bundling fails (LLM-in-a-flash) until layout is globally optimized (Ripple); naive SSD expert offload can raise per-token energy ~12× (arXiv 2508.06978); unstructured pruning saves zero bandwidth without a dense-readable format (Endor).

# 7.3 Verified open intersections (nobody occupies these)

1. Progressive precision as a demand-paged memory hierarchy. Low-bit base resident in unified memory; higher-order residual planes/codebooks on NVMe, fetched per-block conditioned on sensitivity × token need × cache state. §5 encodings + §4 paging: never combined, on any OS.
2. Uncertainty-gated, storage-tiered precision escalation with a correctness story. Base-pass margin (or learned error signal, ρ=0.82 features) decides whether to stream residuals; exactness via rejection sampling or CALM-style calibrated risk; bytes/token as the optimized objective (no §5 or §6 paper measures it). The narrow-but-real gap after excluding QSpec/BiLD/DejaVu near-misses.
3. Incremental refinement kernel: compute ΔY = (ΔW)X against cached activations instead of recomputing the layer (GMRES-IR transposed to inference). Unpublished primitive.
4. Model-aware weight cache on macOS: MRU-for-cyclic-scans, ARC for experts, DBMIN per-tensor-class pools, purgeable MTLHeap warm tier, router-lookahead prefetch at QD≥8 — each ingredient proven in isolation (llama.cpp #18758: 2.23× from layout alone; #20757 open request); the composition doesn't exist, least of all on Metal.
5. Metal kernels for any §5 format (bitplane matvec, MSB-sliced decode, LUT-free trellis) and for any §3 gather kernel (TEAL port). Zero published implementations.
6. Importance-scored residency instead of pruning: Wanda/SparseGPT scores as tier assignment (demoted to Endor-format NVMe, re-materialized on demand), with Sirius-style correction restoring the dense quality ceiling.
7. Residual-aware calibration: optimize a low-bit base jointly with its importance-ranked residual stream under a bytes/token budget (DWQ loop is the on-device vehicle).

# 8. What is settled vs open

Settled (build on, don't re-derive):

• 4-bit weight-only PTQ is a commodity with native Metal speed; 2-bit-effective is the post-training floor for a behavior-preserving base (ParetoQ transition, EfficientQAT ceiling).
• KV-cache quantization on Metal is solved-enough (llama.cpp/mlx-lm, 2–8 bit).
• Dense out-of-core streaming is interactively hopeless: tokens/s ≤ SSD BW ÷ non-resident bytes (ZeRO math, AirLLM, MLX 0.025 tok/s). Selectivity must come first.
• Weight access is predictable where sparsity/routing exists: 0.2 GB/token vs 13.4 GB (LLM in a flash); MoE expert locality + one-layer-lookahead prefetch works.
• Token decisions largely survive cheap approximation: ~91% top-1 agreement at 4-bit; 69–75% acceptance at 25–40% depth; >90% QuantSpec acceptance; rejection sampling repairs the rest exactly.
• Exact-output speculation works on Metal today (ReDrafter/MLX, llama.cpp).
• The paging unit must be chosen by the storage medium: 16 KB-aligned, ≥256 KB effective reads at QD≥8 (expH: 13 GB/s at 1 MiB QD8 vs 67 MB/s at 4 KiB QD1); compute overlap is free (<5% contention).
• Formal worst-case certification of token decisions at 7B+ scale is out of reach; statistical calibration (CALM recipe) is the realistic guarantee class.
• Aggregate accuracy is not a valid quality certificate (flips ≤15% at 1% accuracy delta); use KL / flips / agreement + reasoning tasks.

Open (the research):

• Bytes-read-per-token for any progressive representation — unmeasured in all of §5.
• The joint distribution (base-model margin × full-model agreement) — decides whether escalation is 5% or 40% of tokens (Experiment G).
• Working-set statistics (Jaccard, reuse distance, window-union growth) for thresholded-SwiGLU masks — all published numbers are ReLU-era (Experiments A–C).
• Metal decode cost of bitplane/trellis/gather kernels — does the i-quant penalty apply? (Experiments D/E).
• Whether refinement-relevant blocks are stable across tokens/domains — if not, the ~35–100:1 RAM:SSD ratio kills residual paging (residual traffic must be ≲1–2% of weight bytes/token or fully overlapped).
• The block-deferred verification policy (when to flush uncertain tokens) — SpecExec budgets + Kangaroo confidence-stop are only starting points.
• Whether a 2-bit (vs 4-bit) base keeps agreement high enough — heavy KLD tails (max 22+ nats) say maybe not; measurable cheaply before any engineering.

# 9. Implications for Phase 3

The gap map for research_gaps.md should be organized around the seven open intersections in §7.3, constrained by §1 substrate facts and the §8 open measurements. Three structural conclusions carry over: (1) any candidate must decouple bytes/token before touching the SSD — pure capacity decoupling is disproven; (2) the base-resident + demand-refined shape (low-bit core in unified memory, residuals/cold weights on 16 KB-aligned NVMe blocks read at ≥256 KB QD≥8) is the only regime that helps dense models, and every ingredient for it exists separately with favorable Apple-specific mechanics (zero-copy Metal buffers, purgeable heaps, free compute/I-O overlap) that discrete-GPU prior work could not use; (3) the falsifying measurements are cheap and must precede prototypes — Experiments G (margin×agreement), B/C (SwiGLU working-set stability), D/E (Metal decode cost of progressive layouts), H (already partially done) directly decide which of the §7.3 gaps survive. Phase 3 should generate its ≥20 candidates by crossing the four decoupling axes (precision, sparsity, sharing/delta, speculation) with the four systems levers (residency policy, layout, prefetch, verification), pruning cells already occupied per §7.1.