Phase 2: state-of-the-art map (research/state_of_the_art.md)

~35 technique groups in six families, charter §5 fields, overlap
analysis, settled-vs-open ledger, expH substrate numbers integrated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Simon-Pierre Boucher committed 6 h ago (Aug 12, 2026) parent f5305b3

Showing 2 changed files with +379 and −0

modified research/LOG.md +23 −0

		`@@ -83,3 +83,26 @@ Format per entry: date/time (local, with timezone) · question · experiment ·`
83	83	uncached; distrust any number above the concurrent iostat ceiling; 16 KiB pages.
84	84	- Decision: Prioritize expD (progressive reconstruction) and expG (decision stability)
85	85	after Phases 2–4 paperwork. Sequential-read claims embargoed until a >RAM-size-file rerun.
	86	+
	87	+---
	88	+
	89	+## 2026-08-12 00:50 EDT — Phase 2 complete: state-of-the-art map
	90	+
	91	+- Question: How does the technique landscape organize, where do approaches overlap, and
	92	+ which seemingly-novel ideas are already known?
	93	+- Experiment: Synthesis of the five Phase 1 notes into `research/state_of_the_art.md`
	94	+ (356 lines, ~35 technique groups, six families, all charter §5 fields, overlap analysis,
	95	+ settled-vs-open ledger).
	96	+- Result (load-bearing overlap findings): (1) quantization+offload and precision×tier
	97	+ unification already exist (llama.cpp, FlexGen, M2Cache) — but always statically, never
	98	+ refining a fetched block; (2) sparsity-driven flash paging exists (LLM-in-a-flash,
	99	+ PowerInfer-2, Ripple, DIP) but ReLU-only, never macOS/Metal, always predict-and-hope with
	100	+ no correction; (3) QSpec/QuantSpec prove >90% low/full-bit agreement with exact repair but
	101	+ keep BOTH precisions resident — the unclaimed move is using agreement to avoid LOADING
	102	+ high-precision bytes; (4) the progressive/nested code family never reports bytes/token —
	103	+ the charter's central metric is literally unreported there; (5) several "obvious" ideas are
	104	+ known including failure modes (weights-as-mmap collapse at 0.025 tok/s; DFloat11 decode tax;
	105	+ i-quant LUT penalty on Metal). Seven verified open intersections listed in §7.3.
	106	+- Interpretation: The gap space is real, narrow, and well-bounded — enough for ≥20
	107	+ concrete gap hypotheses without inventing variants of one idea.
	108	+- Decision: Launch Phase 3 (research_gaps.md) now; Phase 4 ranking after review.

added research/state_of_the_art.md +356 −0

		`@@ -0,0 +1,356 @@`
	1	+---
	2	+project: localvm-research
	3	+document: research/state_of_the_art
	4	+author: Simon-Pierre Boucher
	5	+contact: contact@spboucher.ai
	6	+created: 2026-08-12
	7	+status: draft
	8	+---
	9	+
	10	+# State of the Art: Post-Training Execution of Over-Budget LLMs on Apple Silicon
	11	+
	12	+Phase 2 deliverable (charter §5). Synthesizes the five Phase 1 notes
	13	+(`research/notes/{quantization, sparsity_pruning, out_of_core_memory_systems,
	14	+decomposition_progressive, speculation_error_stability}.md`). Every source cited here is
	15	+listed with URL and access date in `research/bibliography.md`. Per-technique blocks use the
	16	+mandated fields; abbreviations: Mod = model modification required, Retrain =
	17	+retraining required, Mem/BW/Compute = memory / bandwidth / compute reduction,
	18	+AS/Metal = works on Apple Silicon (Metal), OSS/macOS = open-source implementation
	19	+and whether it builds on macOS arm64.
	20	+
	21	+Organizing frame (charter §2): `total model size ≠ resident size ≠ bytes read per token ≠
	22	+parameters materially required for this token`. Each taxonomy family below decouples a
	23	+different pair of these quantities — or fails to.
	24	+
	25	+---
	26	+
	27	+## 0. Taxonomy
	28	+
	29	+\| Family \| What it decouples \| Verdict in one line \|
	30	+\|---\|---\|---\|
	31	+\| §1 Substrate facts \| (constraints, not techniques) \| RAM:SSD bandwidth ratio ~35:1 sequential-best-case, ~7000:1 at 4 KiB QD1 — every design follows from this \|
	32	+\| §2 Static compression \| total size ↓ (uniformly) \| Solved to ~4 bpw; frontier at 2–2.5 bpw is CUDA-only; every byte still read every token \|
	33	+\| §3 Sparsity & conditional execution \| params required per token ≪ total \| 85–97% skippable on ReLU models, 40–50% on SwiGLU; no Metal kernels anywhere \|
	34	+\| §4 Out-of-core & paging \| resident size < total size \| Capacity decoupling alone is worthless (AirLLM, MLX-mmap); pays only when combined with §3 selectivity \|
	35	+\| §5 Progressive / nested representations \| bytes read ≠ bytes stored \| Encodings exist (2024–26) and are near-SOTA per operating point; all select the point statically; none pages residuals \|
	36	+\| §6 Speculation, verification & stability \| exact output ≠ exact computation \| 70–90%+ of token decisions survive cheap approximation; exactness recoverable by rejection sampling; formal bounds vacuous at 7B+ \|
	37	+
	38	+---
	39	+
	40	+## 1. Substrate facts: Apple Silicon target (M5 Max, 48 GB, AP2048Z NVMe)
	41	+
	42	+Measured and documented constraints that gate every technique below.
	43	+
	44	+- Unified memory bandwidth: ~460–614 GB/s on M5 Max (Apple specs, config-dependent).
	45	+ CPU and GPU share it — no VRAM/DRAM split, no PCIe. Bandwidth-bound decode ceiling ≈
	46	+ memory BW ÷ resident-active bytes (fully resident Q4-70B ≈ 40 GB → ~11 tok/s ceiling).
	47	+- Internal NVMe (measured, expH, this repo, 2026-08-12): ~13 GB/s peak random read at
	48	+ 1 MiB blocks QD8; 67 MB/s at 4 KiB QD1. That is ~35:1 RAM:SSD at best and ~7000:1
	49	+ worst-case — small random reads are the cliff, not sequential bandwidth. Our peak exceeds
	50	+ the literature's Apple-NVMe priors (LLM in a flash: >6 GiB/s sequential, ~2.25 GiB/s
	51	+ effective sparse on M1 Max), so paging economics are more favorable on our hardware.
	52	+- Concurrent Metal GPU matmul load reduces SSD random-read throughput by <5%
	53	+ (expH, this repo, 2026-08-12) — compute/I-O overlap is essentially free; memory-controller
	54	+ contention is a non-issue.
	55	+- F_NOCACHE does not bypass already-resident pages (expH discovery, this repo,
	56	+ 2026-08-12; consistent with Apple dev forums #25464 and fio #48). There is no O_DIRECT,
	57	+ no posix_fadvise, no io_uring; eviction from the Unified Buffer Cache is opaque
	58	+ approximate-LRU. Cold-state experiments require `purge(8)` + `vm_stat` deltas.
	59	+- Page size is 16 KB — the minimum paging/fault unit; safetensors offsets are not
	60	+ page-aligned (root cause of MLX's mmap dead-end), GGUF's are. `makeBuffer(bytesNoCopy:)`
	61	+ requires page alignment → **any compiled weight format must 16 KB-align every
	62	+ independently pageable block**.
	63	+- Zero-copy path exists and ships: llama.cpp's Metal backend reads mmap'd file pages
	64	+ directly via `MTLResourceStorageModeShared` (discussion #21223) — SSD→page-cache→GPU with
	65	+ no copy, impossible on discrete GPUs. Purgeable `MTLHeap` + residency sets give a
	66	+ three-tier RAM hierarchy (wired-hot / purgeable-warm / evictable-cache) unique to macOS.
	67	+- Wired-memory cliff: `iogpu.wired_limit_mb` defaults to ~66–75% of RAM; unbounded
	68	+ wiring ends in IOGPUMemory kernel panic (mlx-lm #883), not graceful degradation.
	69	+- macOS compressor is useless on weights (quantized weights ≈ incompressible entropy);
	70	+ weight overflow must be file-backed (dropped), never anonymous (compressed/swapped).
	71	+- Decode-kernel cost rule (i-quant lesson, llama.cpp #5617): on Apple GPUs, fewer bits
	72	+ → faster tokens only if decode stays shift/mask-cheap. IQ-quants read ~half the bytes of
	73	+ Q4_0 yet run slower on M2 Max (53.9 vs 63.1 tok/s); LUT-heavy decode is compute-bound
	74	+ on Metal. Bitplanes and bitshift-trellises qualify; big codebook LUTs do not.
	75	+
	76	+---
	77	+
	78	+## 2. Static compression
	79	+
	80	+Every technique here shrinks the checkpoint uniformly; **all bytes of the compressed model
	81	+are still read every token** — memory and bandwidth reductions are proportional and fixed
	82	+at encode time.
	83	+
	84	+2.1 Scalar weight-only PTQ — GPTQ, AWQ, HQQ, SqueezeLLM
	85	+- Problem: checkpoint capacity + proportional decode bandwidth. Mod: re-encoded weights only. Retrain: none (calibration; HQQ calibration-free, 70B in <5 min).
	86	+- Mem: ~4× @4-bit, ~5.3× @3-bit. BW: proportional (GPTQ: 3.25–4.5× generation speedup vs FP16). Compute: neutral (fused dequant). Latency: ∝ bytes.
	87	+- Quality: near-lossless @4-bit (GPTQ OPT-1.3B: FP16 PPL 14.63 → 15.47; naive RTN 48.24). SqueezeLLM 3-bit LLaMA-7B: 18.08 → 7.75 via sensitivity-weighted k-means + 0.05–0.45% FP16 sparse outliers. AWQ more calibration-shift-robust than GPTQ (0.5–0.6 vs 2.3–4.9 PPL under shift).
	88	+- AS/Metal: algorithms platform-neutral; outputs convert to GGUF/MLX. Fast kernels (Marlin/ExLlama/SqueezeLLM) CUDA-only. OSS/macOS: mlx-lm ships GPTQ/AWQ-style recipes natively (yes).
	89	+- Limitation: uniform scalar grids wall below 3 bits. Extension: GPTQ Hessians and SqueezeLLM's dense+sparse split are reusable for allocating residual precision.
	90	+
	91	+2.2 GGUF block formats (llama.cpp k-quants / i-quants / MXFP4)
	92	+- Problem: same, plus per-tensor precision mixing. Mod: convert-time. Retrain: none (i-quants need imatrix calibration).
	93	+- Mem/BW: proportional (Q4_K_M ≈ 4.9 GB for 8B, ~+0.08 PPL). Quality/bit: k-quants beat MLX affine at matched bpw — Q4_K_M 4.88 bpw = 0.0208 nats KL-to-FP16 vs MLX affine 4-bit 0.0577 (Feldman 2026, ~2.8×).
	94	+- AS/Metal: all types have hand-written Metal kernels, but i-quants pay the LUT penalty (§1). OSS/macOS: yes, first-class.
	95	+- Limitation: static per-tensor decisions; no partial loading. Extension: imatrix = ready-made per-block sensitivity signal.
	96	+
	97	+2.3 MLX native quantization + learned quants (affine 2–8 bit; DWQ, dynamic_quant)
	98	+- Problem: Apple-first quantized inference. Mod: convert-time. Retrain: none (DWQ distills quant params only; ≈ +0.6 effective bits).
	99	+- Mem/BW: proportional; MLX 4-bit 8B ≈ 4.5 GB. Latency: within ±10–20% of llama.cpp; MLX wins 4-bit decode, llama.cpp prefill.
	100	+- Quality: affine 4-bit/g64 trails Q4_K_M; g32/DWQ closes gap. AS/Metal: fully native, fused dequant-matmul, quantized-KV. OSS/macOS: yes — the maintained Apple-first PTQ toolchain, pure Python, hackable.
	101	+- Limitation: uniform affine only; no codebooks, no sub-2-bit, no nested layout; precision fixed at convert time. Extension: the `mode` plug-point + custom Metal kernels is where a progressive format would live; DWQ loop is the vehicle for residual-aware calibration.
	102	+
	103	+2.4 Rotation-based W+A quantization — SmoothQuant, QuaRot, SpinQuant
	104	+- Problem: activation outliers blocking W4A4/W8A8. Mod: invariance rotations absorbed offline (output-preserving). Retrain: none (SpinQuant learns rotations, no model retrain).
	105	+- Mem: checkpoint unchanged (weights-side same as 2.1); KV shrinks. BW: mainly KV + prefill. Compute: INT4/8 matmul density — a prefill/throughput play, mostly irrelevant to batch-1 decode BW on Mac.
	106	+- Quality: QuaRot Llama-2-70B W4A4 ≤0.47 PPL loss, 99% zero-shot retention. AS/Metal: no shipped Metal W4A4 path; MLX `quantize_input` is the beginning. OSS/macOS: reference code CUDA.
	107	+- Limitation: wrong bottleneck for us. Extension: incoherence preprocessing is upstream-compatible with any base encoding (QuIP#/QTIP depend on it).
	108	+
	109	+2.5 Codebook / lattice / trellis quantization — QuIP#, AQLM, GPTVQ, QTIP, EXL3, NestQuant
	110	+- Problem: the 2–2.5 bpw frontier. Mod: heavy re-encoding. Retrain: calibration hours; quality depends on fine-tuning (AQLM ~720 GPU-h for 70B).
	111	+- Mem: ~8× (2 bpw usable: QuIP# 70B 2-bit Wiki2 PPL 4.16 vs FP16 3.12; EXL3 70B coherent at 1.6 bpw in <16 GB). BW: conditional on decode cost — QTIP >80% of peak GPU BW (bitshift trellis, ~2 instr/weight); AQLM's 1 MiB codebooks blow L1 (20.6 vs 106.3 tok/s QTIP-class). GPTVQ is the one VQ validated on unified-memory Arm CPU with simultaneous DRAM+latency wins.
	112	+- Quality: SOTA per bit below 3; first regime where 3-bit scales better than 4-bit (QuIP#). AS/Metal: no Metal implementation of any of these exists; i-quant evidence predicts LUT variants lose on Apple GPUs. OSS/macOS: CUDA-only.
	113	+- Limitation: fixed bitrate at encode; CUDA ecosystem. Extension: RVQ/additive structure is inherently progressive (→ §5); a LUT-free Metal trellis decoder is unclaimed.
	114	+
	115	+2.6 Extreme low-bit — BiLLM, BitNet/bitnet.cpp, ParetoQ, EfficientQAT
	116	+- Problem: ≤2-bit regimes. Mod/Retrain: BiLLM PTQ (none); BitNet trained-from-scratch (excluded by charter §14); ParetoQ/EfficientQAT = QAT finetuning.
	117	+- Mem: 8–16×. Quality: BiLLM ~1.08 bpw 70B PPL 8.41 (clearly degraded); EfficientQAT w2g64 7B PPL 6.86 vs 5.47 FP16, 70B −3% acc in 41 A100-h — the realistic "2-bit that still behaves like the original" ceiling. ParetoQ: sharp representational transition between 2 and 3 bits — ≤2-bit leaves the pretrained basin.
	118	+- AS/Metal: bitnet.cpp is CPU-only (NEON LUTs) yet runs a 100B ternary model at 7.45 tok/s on M2 Ultra — existence proof that sub-2-bit makes 100B-class Mac-feasible. OSS/macOS: bitnet.cpp yes; others CUDA/PyTorch.
	119	+- Limitation: pure-PTQ sub-2-bit destroys quality; QAT violates our budget. Extension: fixes the design floor — a base representation must be ≥2-bit-effective (safer 2.5–3) to preserve routing/decision structure without retraining.
	120	+
	121	+2.7 KV-cache quantization — KIVI, KVQuant, Coupled Quantization
	122	+- Problem: KV bytes/token rival weight bytes at long context. Mod: runtime cache format. Retrain: none.
	123	+- Mem/BW: KIVI 2-bit (K per-channel, V per-token) 2.6× peak-mem; KVQuant 3-bit <0.1 PPL, 1M-token context on one A100; CQ ~1 bit/channel-equiv.
	124	+- AS/Metal: the solved part of the working-set problem on Macs — llama.cpp `--cache-type-k/v` with flash attention on Metal; mlx-lm `--kv-bits {2,4,8}`; TurboQuant Metal kernels at 3.25/4.25 bits landing. OSS/macOS: yes.
	125	+- Limitation/Extension: any weight working-set argument must co-model KV; asymmetric K/V precision (KVSplit) already pays on Metal.
	126	+
	127	+2.8 SVD family & structural decomposition — SVD-LLM, SliceGPT, LASER
	128	+- Problem: low-rank re-representation. Mod: factorized/sliced weights. Retrain: none (calibration/whitening).
	129	+- Mem/BW: parameter-ratio-proportional, dense (plain GEMMs). Quality: weights are not globally low-rank — SVD-LLM 7.73 PPL vs ASVD 11.14 at 20% compression (LLaMA-7B), but ≥40% costs large downstream accuracy. SliceGPT: −25% params at 99% zero-shot (Llama-2-70B), up to 1.55× throughput — the only pruning-family transform that trivially runs fast on Metal (smaller dense GEMMs). LASER: replacing selected late-MLP matrices with low-rank improves accuracy up to +30 pts — high-order components of specific matrices are noise-like.
	130	+- AS/Metal: excellent (dense matmul; MLX-native, composable with affine quant). OSS/macOS: yes (PyTorch, ports trivial).
	131	+- Limitation: fixed rank at compile; residual discarded despite carrying most spectral energy. Extension: low-rank resident hot path + SSD-resident residual — none of these papers stores the residual at all.
	132	+
	133	+2.9 Low-rank + quantized residual — CALDERA
	134	+- Problem: 2–2.5 bpw via `W ≈ Q + LR` with error bounds. Mod: re-encode. Retrain: calibration-aware alternating minimization.
	135	+- Mem: 2–2.5 bpw class. Quality: competitive at extreme ratios. AS/Metal: LR is dense (fine); Q uses QuIP#-style lattices (CUDA). OSS/macOS: no.
	136	+- Limitation: single operating point; joint optimization couples terms — dropping one is not quality-graceful. Extension: re-derive with a progressivity constraint (base alone usable) — the bridge to §5.
	137	+
	138	+2.10 Cross-layer sharing & delta compression — Basis Sharing, DeltaLLM, Relaxed Recursive Transformers, BitDelta, Delta-CoMe
	139	+- Problem: redundancy between matrices. Mod: shared trunk + per-layer corrections. Retrain: light (DeltaLLM ~30–40M tokens; RRT distillation; BitDelta minutes).
	140	+- Mem: modest as compression (12–25%); as evidence enormous: finetune deltas quantize to ~1 bit/param near-losslessly (BitDelta; Delta-CoMe holds on math/code) — proof that "model = reference + extremely compressible correction" is real structure. BitDelta iterated on itself yields a monotone stack of 1-bit masks → an accidental progressive code.
	141	+- AS/Metal: sign-mask + scale decode is trivially Metal-friendly; Basis Sharing/DeltaLLM are dense ops. OSS/macOS: PyTorch, portable.
	142	+- Limitation: zero-shot layer tying degrades; ratios far from 10×. Extension: shared trunk as cache-resident core, deltas as SSD pages — untried as a memory-hierarchy assignment.
	143	+
	144	+2.11 Lossless & generative recompression — DFloat11, SeedLM
	145	+- DFloat11 (lossless Huffman of BF16 exponents): 30% size cut, bit-identical, but 2–3× slower tokens/s than uncompressed-in-VRAM — lossless coding shrinks capacity, not effective bandwidth; entropy decode sits on the critical path. CUDA-only. The cautionary datapoint for any decode-on-load design.
	146	+- SeedLM (Apple): weights → LFSR seed + coefficients; regenerate at inference — trades bandwidth for free compute, exactly the right trade on BW-bound hardware (~4× FPGA at 70B, data-free, 4-bit parity). No Metal port. Extension: "weights need not be stored, only recoverable" — Apple-authored, conceptually adjacent to our thesis.
	147	+
	148	+2.12 One-shot & structured pruning — SparseGPT, Wanda, ShortGPT, Sheared-LLaMA, Minitron
	149	+- Problem: permanent weight removal. Mod: destructive. Retrain: none (one-shot) to heavy (Sheared/Minitron).
	150	+- Quality: 50% unstructured nearly free only at ≥65B (LLaMA-65B 3.56→4.57; 7B 5.68→7.26, ~27% worse); 2:4 much worse at small scale (7B → 11.02). ShortGPT: middle-late layers highly redundant; "Secretly Linear": Procrustes linearity ≈0.99 between consecutive layers.
	151	+- AS/Metal: Apple GPUs have no sparse tensor cores — unstructured sparsity yields zero BW savings unless the representation skips loads (CSR indexing overhead cancels it — Endor's point). OSS/macOS: algorithms portable; speedups are not.
	152	+- Limitation: answers the wrong question (total size, not residency), destroys quality at 7B scale. Extension: importance scores as residency policy — Wanda's \|W\|·‖x‖ is cheap enough to rank weights for RAM-vs-NVMe instead of deletion; Block Influence for layer-level tiers; Endor's bitmap format for dense-readable demoted weights.
	153	+
	154	+---
	155	+
	156	+## 3. Sparsity & conditional execution
	157	+
	158	+These decouple parameters required per token from total parameters — the precondition
	159	+for any paging win (§4).
	160	+
	161	+3.1 Contextual sparsity — DejaVu, ShadowLLM
	162	+- Problem: which heads/neurons does this token need. Mod: none to the model; per-layer MLP predictors added. Retrain: predictors only.
	163	+- Mem: none (model stays resident). BW: up to 85% contextual sparsity ⇒ proportional HBM traffic cut; >2× vs FasterTransformer on OPT-175B, no measured quality loss. ShadowLLM: one early-layer predictor shadows all layers, >15% accuracy over DejaVu criteria at equal sparsity, +20% speed — and gives maximum prefetch lead time.
	164	+- AS/Metal: ReLU-era OPT models; multi-A100 assumptions; no Metal kernels. OSS/macOS: CUDA.
	165	+- Limitation: ReLU-dependent; model must fit in accelerator memory. Extension: "slowly changing hidden states" as an NVMe prefetch signal rather than a FLOP-skip signal.
	166	+
	167	+3.2 ReLUfication — ReLU Strikes Back, ProSparse, TurboSparse, Q-Sparse
	168	+- Problem: SwiGLU killed exact zeros. Mod: activation swap. Retrain: yes (fine-tuning to ~150B tokens) — violates post-training-only, but checkpoints (ProSparse-LLaMA, TurboSparse-Mistral/Mixtral, Bamboo) are downloadable test vehicles.
	169	+- Numbers: 87.9–89.3% sparsity at parity (ProSparse); TurboSparse-Mistral-7B activates 2.5B params/token, Mixtral-47B activates 4.3B. Sets the recoverable ceiling (~85–90%).
	170	+
	171	+3.3 Training-free SwiGLU sparsity — CATS, TEAL, GRIFFIN, DIP
	172	+- Problem: sparsity on stock modern models, no retraining. Mod: none. Retrain: none (calibration thresholds).
	173	+- BW: CATS 50% FFN sparsity at ~99% task retention (≈25% model-wide); TEAL 40–50% model-wide (all matrices) with minimal degradation, 1.53–1.8× decode via Triton; GRIFFIN "flocking" — 50% of FF params selected once per sequence (prefetch-friendly granularity); DIP: cache-aware masks — sparsity chosen given DRAM contents: 46% less memory, 40% more throughput, <0.1 PPL on Phi-3-Medium under mobile limits — the first explicit "sparsity-as-cache-policy" formulation.
	174	+- Quality caveat (Sirius, NeurIPS 2024): contextual sparsity degrades reasoning/GSM8K/coding specifically; correcting ~11% of tokens by dense verification restores accuracy at ~78% of the efficiency gain. Perplexity-based "99% retention" claims overstate quality.
	175	+- AS/Metal: all kernels CUDA (Triton); no Metal equivalents exist anywhere. OSS/macOS: no.
	176	+- Limitation: 40–50% ≠ 97% — a 2× bytes/token cut, not 50×. Extension: TEAL as the first Metal port target; thresholded-SwiGLU masks as paging policy — bytes/token for this has never been published; working-set stats (Jaccard, reuse distance) exist only for ReLU models → our Experiments A–C.
	177	+
	178	+3.4 SparQ Attention (attention-side analogue)
	179	+- Fetch only KV rows whose keys matter for the current query: up to 8× attention data-transfer cut, negligible loss, no fine-tuning. Demand-driven fetching works for state, not just weights; complements §2.7.
	180	+
	181	+---
	182	+
	183	+## 4. Out-of-core & paging systems
	184	+
	185	+4.1 Dense streaming — FlexGen, ZeRO-Inference, AirLLM
	186	+- Problem: capacity only. Mod/Retrain: none. Mem: resident ≈ 0 possible. BW: bytes/token = full checkpoint unless batched — the hard ceiling: tokens/s ≤ storage BW ÷ non-resident bytes. On our SSD a 70 GB non-resident model can never exceed ~0.1–0.2 tok/s dense.
	187	+- Latency: FlexGen OPT-175B on 16 GB GPU ≈ 1 tok/s effective only at huge batch; AirLLM 70B in 4 GB, 15–30 min/response. AS/Metal: CUDA (FlexGen LP idea portable). Verdict: proof that capacity decoupling alone is worthless interactively; these are the harness straw-man controls.
	188	+
	189	+4.2 LLM in a flash (Apple, ACL 2024) — closest prior art, ran on our platform
	190	+- Mod: predictors added (<2.4% overhead); ReLU-family models only. Retrain: predictors only.
	191	+- Mem: DRAM ≈ 52% of model (attention+embeddings pinned; FFN on flash). BW: bytes/token 13.4 GB naive → 0.2 GB with predictor+windowing (OPT-6.7B, k=4 window; 2.4–3.1% of FFN neurons touched/token) — the single most important measured number in the literature for charter §8.3.
	192	+- Latency: I/O per token 2196 ms → 105 ms on M1 Max; 4.23× Metal/M1, 7.44× Metal/M2 Ultra; runs models ~2× DRAM. Effective flash throughput ~2.25 GiB/s with row-column bundling at ≥32 KiB × 32 threads.
	193	+- AS/Metal: yes — measured on M1 Max / M2 Ultra with Metal. OSS/macOS: no code released; nothing in MLX/llama.cpp implements it.
	194	+- Limitation: ReLU-only (SwiGLU "not addressed"); attention never paged; ≤2× oversubscription; fp16 bundles (no quantization co-design); negative result: co-activation bundling failed (hot neurons re-loaded — later solved by Ripple's global placement).
	195	+- Extension: reproduce its throughput surface on our AP2048Z (expH); redo for SwiGLU/MoE with 4-bit bundles and Metal compute.
	196	+
	197	+4.3 PowerInfer / PowerInfer-2 / Ripple
	198	+- PowerInfer (SOSP 2024): power-law hot/cold neuron split, hot→VRAM, cold→CPU; 13.2 tok/s OPT-175B-class on one 4090. GPU/CPU split is meaningless on unified memory; Mac analogue is RAM/NVMe. macOS status verified: CPU-only, "limited" gains; Metal sparse backend planned, never shipped.
	199	+- PowerInfer-2: 47B (TurboSparse-Mixtral) at 11.68 tok/s on a 24 GB phone over UFS 4.0 (~1 GB/s random) — the closest architectural template for a Mac (single memory pool + flash). Design lessons: segmented neuron cache; Gate-first two-phase reads; predictors cost 2.6 GB DRAM at 47B; requires ReLUfied models; P99 +40.9%. Our NVMe is ~4–13× faster than its storage — economics transfer favorably; nobody has built it.
	200	+- Ripple: co-activation-aware flash layout (offline global placement) — belongs in our compile stage.
	201	+- OSS/macOS: PowerInfer yes-but-CPU-only; PowerInfer-2 closed, Android.
	202	+
	203	+4.4 M2Cache
	204	+- Neuron-level mixed precision × tier unified: hot neurons FP16 in HBM, colder quantized in DRAM, coldest on SSD; up to 14× vs baseline offload, 70B on 24 GB VRAM. The closest published architecture to charter §13 — but Linux/CUDA, PCIe assumptions, static importance ranking, and no refinement: a neuron is fetched at one precision, never upgraded. OSS/macOS: no.
	205	+
	206	+4.5 llama.cpp mmap + Metal (the working macOS evidence)
	207	+- Mod/Retrain: none. Mechanism: GGUF mmap'd; Metal reads file pages zero-copy; MoE overflow works by OS demand paging.
	208	+- Measured (discussion #18758, M5 Pro/AP1024Z, Qwen3-Next-80B): replacing fault-streaming with layout-aware explicit slice reads: 1418 → 370 reads/token, 2.23× cold-decode I/O, +13–14% end-to-end; mmap beat direct I/O because the page cache retains the hot expert working set.
	209	+- Limitation: replacement policy is the kernel's (opaque LRU, scan-vulnerable); no model-aware prefetch even though the router decision is known before the expert runs; two-tier expert cache is an open, unfilled feature request (#20757). OSS/macOS: yes, first-class.
	210	+
	211	+4.6 MLX status
	212	+- Lazy loading only; no paging mechanism at all. Community mmap prototype: 70 GB model on 64 GB → 0.025 tok/s (kernel LRU vs cyclic dense scans = pathological worst case). This negative result defines our problem. Wired-limit/residency-set controls exist; everything else must be built by us.
	213	+
	214	+4.7 MoE offloading — Eliseev & Mazur, cache-conditional experts, MoBiLE, SolidAttention
	215	+- Eliseev–Mazur: LRU expert cache + speculative expert prefetch (apply layer k+1's gate to layer k's hidden state) → Mixtral-8x7B at 2–3 tok/s on 11–16 GB GPUs. "Branch prediction for weights" in embryo; reuse/prefetchability confirmed general (arXiv 2511.05814).
	216	+- Cache-conditional experts: router biased toward resident experts — the model adapts to the memory system. MoBiLE: misses served by smaller substitute experts — never block compute. SolidAttention (FAST 2026): KV on SSD; paging unit chosen by the storage medium, not model granularity (3.1×, 98% KV memory cut at 128k).
	217	+- OS/DB imports with no LLM instantiation (from notes §4): MRU for cyclic weight scans (DBMIN 1985 — LRU is worst-case for dense decode, MRU is the theoretical floor; apparently never applied); ARC/ghost-list expert caches; per-tensor-class buffer pools; anti-caching's "never block on a miss"; Pythia-style learned prefetch; working-set admission control (Denning); purgeable-MTLHeap kernel-cooperative caches; five-minute-rule residency economics.
	218	+
	219	+---
	220	+
	221	+## 5. Progressive / nested representations ⭐
	222	+
	223	+The family that directly attacks `bytes stored ≠ bytes read`. All of it is 2024–2026.
	224	+
	225	+5.1 Any-Precision LLM (ICML 2024 oral) / Matryoshka Quantization (ICLR 2025 oral)
	226	+- Problem: one artifact, many precisions. Mod: bitplane / MSB-nested re-encoding. Retrain: Any-Precision none (incremental upscaling from a 3-bit seed, <1 min for 7B); MatQuant QAT or OmniQuant-style PTQ.
	227	+- Mem: {3..8}-bit Llama-2-7B in 8.4 GB vs 29.9 GB separate (3.56×). BW: bytes/token = f(precision requested), decoupled from storage — "reduced bit-width directly translates into proportional speedup, as we simply load the specified number of bits." Quality: each slice matches dedicated models; MatQuant int2 slice up to +10% over dedicated int2 QAT; int2-FFN Gemma-2 9B beats int8-FFN Gemma-2 2B.
	228	+- AS/Metal: none — engines are CUDA (bitplane layout, bit-transpose kernels); bitplane decode is bit-manipulation-heavy, Apple-GPU cost unknown (→ Experiment D/E). OSS/macOS: no.
	229	+- Limitation: precision scheduled globally and statically; full chosen precision loaded for every weight every token; all planes kept resident.
	230	+
	231	+5.2 BitStack (ICLR 2025) / RRQ / Drop-by-Drop
	232	+- BitStack: training-free iterative significance-weighted decomposition → ~1-bit residual blocks, universally sorted by importance; runtime loads as many as memory allows → megabyte-granular quality↔residency tradeoff, matching/beating GPTQ/AWQ at extreme ratios. Known weakness: on-the-fly reconstruction slows inference — reconstruction must be fused into the matmul.
	233	+- RRQ (2026): calibration-free 2-bit base + stacked 2-bit RTN residuals; multi-precision package for Qwen3-8B in ~1293 s. Drop-by-Drop (2026): AQLM-style codebooks with Matryoshka supervision, grounded in successive-refinement theory; ordered codebooks droppable at inference; explicitly motivates "hardware-aware dynamic loading" — the closest published object to a progressive residual base representation. No systems implementation, no SSD tier, no Apple port.
	234	+- BitStack is an unwitting rediscovery of embedded wavelet coding (EZW/SPIHT/EBCOT); nobody has connected weight encodings to R-D-optimal per-block truncation, Nanite-style selective residency, or GMRES-IR-style residual correction (mature machinery in graphics/HPC/DB fields).
	235	+
	236	+5.3 Phase-/token-adaptive precision — PMPD, QuickSilver, FlexQuant, DP-LLM, MoBiQuant
	237	+- PMPD (ICLR 2025): precision lowered as generation deepens, task/prompt-adaptive schedulers; 3.8–8.0× decode on an NPU — validates that required precision is position-dependent and runtime-schedulable. QuickSilver/FlexQuant/DP-LLM/MoBiQuant: entropy/sensitivity-gated per-token bit-width over Matryoshka overlays.
	238	+- All are feed-forward heuristics: nothing verifies the low-precision tokens afterward — lossy, empirical-only. And all keep every precision level resident. Confirms the mechanism is implementable; none closes the loop with exactness or with storage.
	239	+
	240	+Common ledger for §5: no member selects precision per weight-block per token; none pages residuals from SSD; none reports bytes-read-per-token (they report resident size); none has Metal kernels.
	241	+
	242	+---
	243	+
	244	+## 6. Speculation, verification & decision stability
	245	+
	246	+6.1 Classical + head-based speculative decoding — Leviathan/Chen, Medusa, EAGLE-3, ReDrafter
	247	+- Problem: amortize expensive-model reads over multiple tokens. Mod: none (draft model) or trained heads. Retrain: heads only.
	248	+- Exactness: rejection sampling provably preserves the target distribution — the canonical proof that approximate computation + cheap correction = exact computation. Numbers: 2–3× (Leviathan; greedy > sampling — argmax easier to reproduce); EAGLE-3 3.0–6.5×, acceptance length up to 7.5; production α ≈ 0.6–0.8.
	249	+- AS/Metal: verified on Apple hardware — ReDrafter on MLX: 1.37× M1 Max, 2.3× M2 Ultra; llama.cpp `--model-draft` and mlx_lm ship speculation on Metal. OSS/macOS: yes.
	250	+- Limitation: verification granularity = whole target model over a token block.
	251	+
	252	+6.2 Depth self-speculation — Draft&Verify, LayerSkip, Kangaroo, SWIFT, CLaSp
	253	+- Draft = same weights, fewer layers; verify = all layers; lossless. 1.2–2.16×. Measured acceptance: 68.9% (Llama-2-7B drafting at layer 8/32), 74.5% (13B at 15/40); CLaSp: Llama-3-70B tolerates skipping 44/80 layers. Tuned Lens "prediction depth": many tokens stabilize well before the final layer. Retrain: LayerSkip yes; SWIFT/CLaSp none. AS/Metal: control flow, trivially portable. Limitation: the draft axis is depth, never precision-of-weights-with-storage-tier.
	254	+
	255	+6.3 Precision self-speculation — QSpec, QuantSpec, ML-SpecQD
	256	+- Draft and target share the same weights at different precisions: QSpec W4A4 draft / W4A16 verify, 1.64×, no quality loss, no training; QuantSpec (Apple+Berkeley, ICML 2025): 4-bit hierarchical-KV+weights draft, full-precision verify, >90% acceptance, ~2.5×, edge-motivated; ML-SpecQD: MXFP4 cast as turnkey draft, >2×.
	257	+- This proves the core premise: a 4-bit cast of the same model agrees with the 16-bit model on the large majority of tokens, and disagreements are exactly repairable by shared-weight verification. What none do: exploit agreement to avoid storing/loading high-precision weights — both operand sets stay resident; the win is arithmetic, not capacity or bytes/token. OSS/macOS: CUDA.
	258	+
	259	+6.4 Speculation × offloading — SpecInfer, SpecExec
	260	+- SpecInfer: 2.6–3.5× specifically for offloading-based inference. SpecExec: huge draft trees verified by an offloaded 70B in one pass — Llama-2-70B at 4–6 tok/s (4-bit) on consumer GPUs, 10.6–18.7× over sequential offloaded decoding, ~20 accepted tokens per full-model weight sweep. The key economics result: when reading the model costs seconds, speculation divides bytes/token by the acceptance length. OSS/macOS: CUDA, concepts portable.
	261	+
	262	+6.5 Confidence-gated escalation (lossy) — BiLD, cascades, CALM
	263	+- BiLD: small model decodes; fallback/rollback to the large model on low confidence — 2.12×, no exactness guarantee. FrugalGPT/RouteLLM: ~1 in 6 queries needs the big model. CALM (NeurIPS 2022): per-token early exit with distribution-free risk calibration — sequence-level consistency with the full model provably maintained at chosen ε; ~3× decode. The strongest published answer to "can computation stop when more precision can't change the output" — answered statistically, for the depth axis only. Nobody has done CALM for weight precision / residual count.
	264	+
	265	+6.6 Formal error bounds (§4.9 verdict: vacuous at scale)
	266	+- Self-attention is not globally Lipschitz (Kim et al. 2021); local constants grow ~√n in sequence length (Castin et al.) and compose to astronomically loose logit bounds over 30–80 layers. CROWN/auto_LiRPA supports weight perturbations in principle but is O(m²n³) — BERT-small scale; DeepT certifies a few layers in minutes. 4–5 orders of magnitude from a 7B decoder.
	267	+- Usable primitives anyway: the margin certificate (argmax stable iff top-2 margin > 2·‖Δlogits‖∞ bound) is trivial once any bound exists; QEP (NeurIPS 2025): layer-wise quantization errors compound near-exponentially with depth; EMNLP 2025: residual-stream magnitudes predict which inputs break quantization (ρ = 0.82) → a cheap learned per-input error predictor + CALM calibration is the realistic instrument. GMRES-IR (Carson–Higham): the canonical "factorize cheap, refine with residuals, guaranteed" pattern — never written down for transformer inference; probabilistic error analysis (√n·u constants) replaces vacuous worst cases.
	268	+
	269	+6.7 Measured decision stability (the §4.10 evidence base)
	270	+- Same-top-token at 4-bit-class quantization: 90.87–91.23% (llama.cpp KLD tooling; mean KLD ≈ 0.054, 99th pct 0.53, max ~22–26 — heavy tail). MLX: mean KLD 0.014 (8-bit) → 0.059–0.113 (4-bit variants); MoE router protection changes rankings — which weights get precision matters more than average bpw.
	271	+- "Accuracy is Not All You Need" (Microsoft): models within 1% aggregate accuracy flip up to ~15% of individual answers (25%+ for layer-dropping); flips correlate with KL at Spearman 0.96–0.97 → aggregate benchmarks cannot certify a compressed runtime; KL/flips/agreement are the quality currency (charter §8.2).
	272	+- Fragility floor: batch-size-dependent rounding alone (~1e-5 relative) diverges greedy completions (Thinking Machines) — a persistent 5–15% of tokens sit on a knife's edge, and they are disproportionately the load-bearing ones. The joint distribution (cheap-model margin × agreement with full model) has never been published — Experiment G fills a real gap.
	273	+
	274	+---
	275	+
	276	+## 7. Overlap analysis
	277	+
	278	+### 7.1 Combinations already tried (do not reinvent)
	279	+
	280	+\| Combination \| Where it exists \| Status \|
	281	+\|---\|---\|---\|
	282	+\| Quantization + offload \| FlexGen (4-bit + LP placement); llama.cpp (GGUF + mmap + Metal zero-copy); ZeRO-Inference (+quantization) \| Shipped, mature. Static composition only \|
	283	+\| Quantization + tiering unified per neuron \| M2Cache (precision and tier as one axis) \| Exists (CUDA); no refinement, static ranking \|
	284	+\| Sparsity + flash paging \| LLM in a flash; PowerInfer-2; Ripple; DIP (cache-aware masks); VLM-in-a-flash \| Exists — ReLU-family only, Android/no-code, never macOS/Metal, never with correction \|
	285	+\| Sparsity + quality correction \| Sirius (11% dense-verified tokens restore reasoning) \| Exists (CUDA); never composed with paging \|
	286	+\| Precision-speculation (same weights, two precisions) \| QSpec, QuantSpec, ML-SpecQD \| Exists; both precisions resident; goal is arithmetic speed, not bytes/token \|
	287	+\| Speculation + offloading \| SpecInfer, SpecExec (~20 tokens per weight sweep) \| Exists (CUDA); draft is a separate model, not a precision prefix of the target \|
	288	+\| Depth-drafting + exact verification \| Draft&Verify, LayerSkip, SWIFT, CLaSp \| Exists, portable; depth axis only \|
	289	+\| Nested precision + per-token scheduling \| PMPD, QuickSilver, DP-LLM, MoBiQuant \| Exists; feed-forward heuristic, no verification, no storage tier \|
	290	+\| Expert prefetch by hidden-state lookahead \| Eliseev–Mazur; Fate; SpecMD (Apple); pre-gated MoE \| Exists; MoE-only (discrete routable units); no dense-model analogue \|
	291	+\| Router/cache co-design \| Cache-conditional experts; MoBiLE (substitute-on-miss) \| Exists; MoE-only \|
	292	+\| Low-rank + quantized residual \| CALDERA; SqueezeLLM dense+sparse; BiLLM binary residual \| Exists as static encodings; never as a runtime load order \|
	293	+
	294	+### 7.2 Seemingly novel ideas that are actually known
	295	+
	296	+- "Treat weights like virtual memory" — llama.cpp mmap has done it since 2023, and the MLX prototype measured its failure mode (0.025 tok/s): known, including why naive versions die (kernel LRU × cyclic scans).
	297	+- "Hot/cold weight split by activation frequency" — PowerInfer's power-law statistics (SOSP 2024).
	298	+- "Escalate to the expensive model only when uncertain" — BiLD (2023), cascades, CALM (with calibrated guarantees, depth axis).
	299	+- "Draft with a cheap version of the same model, verify exactly" — QSpec/QuantSpec own "cheap numerics, exact verify"; LayerSkip owns "cheap depth, exact verify".
	300	+- "Choose the sparsity mask based on what's already cached" — DIP (Qualcomm, 2024).
	301	+- "Store base + refinable residuals of weights" — Any-Precision/MatQuant/BitStack/RRQ/Drop-by-Drop; also EZW/JPEG2000 thirty years ago.
	302	+- "Quality-vs-resident-size as a runtime knob" — BitStack demonstrates it (statically) end to end.
	303	+- Known negative results to respect: lossless decode on the critical path costs 2–3× (DFloat11); LUT-heavy decode loses on Metal (i-quants); naive co-activation bundling fails (LLM-in-a-flash) until layout is globally optimized (Ripple); naive SSD expert offload can raise per-token energy ~12× (arXiv 2508.06978); unstructured pruning saves zero bandwidth without a dense-readable format (Endor).
	304	+
	305	+### 7.3 Verified open intersections (nobody occupies these)
	306	+
	307	+1. Progressive precision as a demand-paged memory hierarchy. Low-bit base resident in unified memory; higher-order residual planes/codebooks on NVMe, fetched per-block conditioned on sensitivity × token need × cache state. §5 encodings + §4 paging: never combined, on any OS.
	308	+2. Uncertainty-gated, storage-tiered precision escalation with a correctness story. Base-pass margin (or learned error signal, ρ=0.82 features) decides whether to stream residuals; exactness via rejection sampling or CALM-style calibrated risk; bytes/token as the optimized objective (no §5 or §6 paper measures it). The narrow-but-real gap after excluding QSpec/BiLD/DejaVu near-misses.
	309	+3. Incremental refinement kernel: compute ΔY = (ΔW)X against cached activations instead of recomputing the layer (GMRES-IR transposed to inference). Unpublished primitive.
	310	+4. Model-aware weight cache on macOS: MRU-for-cyclic-scans, ARC for experts, DBMIN per-tensor-class pools, purgeable MTLHeap warm tier, router-lookahead prefetch at QD≥8 — each ingredient proven in isolation (llama.cpp #18758: 2.23× from layout alone; #20757 open request); the composition doesn't exist, least of all on Metal.
	311	+5. Metal kernels for any §5 format (bitplane matvec, MSB-sliced decode, LUT-free trellis) and for any §3 gather kernel (TEAL port). Zero published implementations.
	312	+6. Importance-scored residency instead of pruning: Wanda/SparseGPT scores as tier assignment (demoted to Endor-format NVMe, re-materialized on demand), with Sirius-style correction restoring the dense quality ceiling.
	313	+7. Residual-aware calibration: optimize a low-bit base jointly with its importance-ranked residual stream under a bytes/token budget (DWQ loop is the on-device vehicle).
	314	+
	315	+---
	316	+
	317	+## 8. What is settled vs open
	318	+
	319	+Settled (build on, don't re-derive):
	320	+- 4-bit weight-only PTQ is a commodity with native Metal speed; 2-bit-effective is the post-training floor for a behavior-preserving base (ParetoQ transition, EfficientQAT ceiling).
	321	+- KV-cache quantization on Metal is solved-enough (llama.cpp/mlx-lm, 2–8 bit).
	322	+- Dense out-of-core streaming is interactively hopeless: tokens/s ≤ SSD BW ÷ non-resident bytes (ZeRO math, AirLLM, MLX 0.025 tok/s). Selectivity must come first.
	323	+- Weight access is predictable where sparsity/routing exists: 0.2 GB/token vs 13.4 GB (LLM in a flash); MoE expert locality + one-layer-lookahead prefetch works.
	324	+- Token decisions largely survive cheap approximation: ~91% top-1 agreement at 4-bit; 69–75% acceptance at 25–40% depth; >90% QuantSpec acceptance; rejection sampling repairs the rest exactly.
	325	+- Exact-output speculation works on Metal today (ReDrafter/MLX, llama.cpp).
	326	+- The paging unit must be chosen by the storage medium: 16 KB-aligned, ≥256 KB effective reads at QD≥8 (expH: 13 GB/s at 1 MiB QD8 vs 67 MB/s at 4 KiB QD1); compute overlap is free (<5% contention).
	327	+- Formal worst-case certification of token decisions at 7B+ scale is out of reach; statistical calibration (CALM recipe) is the realistic guarantee class.
	328	+- Aggregate accuracy is not a valid quality certificate (flips ≤15% at 1% accuracy delta); use KL / flips / agreement + reasoning tasks.
	329	+
	330	+Open (the research):
	331	+- Bytes-read-per-token for any progressive representation — unmeasured in all of §5.
	332	+- The joint distribution (base-model margin × full-model agreement) — decides whether escalation is 5% or 40% of tokens (Experiment G).
	333	+- Working-set statistics (Jaccard, reuse distance, window-union growth) for thresholded-SwiGLU masks — all published numbers are ReLU-era (Experiments A–C).
	334	+- Metal decode cost of bitplane/trellis/gather kernels — does the i-quant penalty apply? (Experiments D/E).
	335	+- Whether refinement-relevant blocks are stable across tokens/domains — if not, the ~35–100:1 RAM:SSD ratio kills residual paging (residual traffic must be ≲1–2% of weight bytes/token or fully overlapped).
	336	+- The block-deferred verification policy (when to flush uncertain tokens) — SpecExec budgets + Kangaroo confidence-stop are only starting points.
	337	+- Whether a 2-bit (vs 4-bit) base keeps agreement high enough — heavy KLD tails (max 22+ nats) say maybe not; measurable cheaply before any engineering.
	338	+
	339	+---
	340	+
	341	+## 9. Implications for Phase 3
	342	+
	343	+The gap map for `research_gaps.md` should be organized around the seven open intersections
	344	+in §7.3, constrained by §1 substrate facts and the §8 open measurements. Three structural
	345	+conclusions carry over: (1) any candidate must decouple bytes/token before touching the
	346	+SSD — pure capacity decoupling is disproven; (2) the base-resident + demand-refined shape
	347	+(low-bit core in unified memory, residuals/cold weights on 16 KB-aligned NVMe blocks read
	348	+at ≥256 KB QD≥8) is the only regime that helps dense models, and every ingredient for it
	349	+exists separately with favorable Apple-specific mechanics (zero-copy Metal buffers,
	350	+purgeable heaps, free compute/I-O overlap) that discrete-GPU prior work could not use;
	351	+(3) the falsifying measurements are cheap and must precede prototypes — Experiments G
	352	+(margin×agreement), B/C (SwiGLU working-set stability), D/E (Metal decode cost of
	353	+progressive layouts), H (already partially done) directly decide which of the §7.3 gaps
	354	+survive. Phase 3 should generate its ≥20 candidates by crossing the four decoupling axes
	355	+(precision, sparsity, sharing/delta, speculation) with the four systems levers (residency
	356	+policy, layout, prefetch, verification), pruning cells already occupied per §7.1.
	357