SPB Git

spb/localvm-research Public License

Running LLMs larger than memory on a consumer Mac — falsification-driven research: margin-gated deferred refinement, out-of-core verification on Apple Silicon. TR-01 published.

Python 63.2% JavaScript 23.5% CSS 11.8% Shell 0.9% Makefile 0.5%
41.0 KB
View source Raw Blame History

# project: localvm-research document: research/notes/decomposition_progressive author: Simon-Pierre Boucher contact: contact@spboucher.ai created: 2026-08-11 status: draft

# Model Decomposition (§4.5) & Progressive / Approximate Computation (§4.6)

Reading notes for charter sections 4.5 and 4.6. Scope: can pretrained weights be re-represented as base + residual, shared component + per-layer correction, or low-rank + sparse/quantized residual — and can computation itself be made progressive, so that quality scales with bytes loaded rather than being fixed at compile time. All quality numbers below are taken from the cited papers, not reproduced locally. Access date for all sources: 2026-08-11.

# 1. Landscape

The literature splits into four families that almost never talk to each other:

  1. Static decomposition (SVD family, structural slicing, Kronecker/tensor-train, codebook VQ): transform the checkpoint once, into a fixed-size smaller representation. State of the art is good at 3–4 bits/param equivalent, usable at 2 bits, and degrades sharply below that. None of these change the runtime contract: every byte of the compressed model is still read for every token.

  2. Sharing/delta decomposition (cross-layer sharing, DeltaLLM, delta compression of finetunes): exploit redundancy between matrices — between layers of one model, or between a base model and its finetunes. The measured compressibility here (1-bit deltas that are near-lossless) is the strongest published evidence that large fractions of transformer weight information are redundant relative to a reference.

  3. Progressive / nested representations (Any-Precision LLM, Matryoshka Quantization, BitStack, recurrent residual quantization, Matryoshka-style additive codebooks): one stored artifact yields many operating points; lower precision is a strict prefix/subset of higher precision. This is the family closest to the localvm-research thesis, and it is young (2024–2026).

  4. Adaptive computation (early exit, dynamic depth, adaptive halting, approximate matmul, mixed-precision iterative refinement, error-bounded lossy compression): vary the amount of computation or precision per input, sometimes with provable error control. Almost all of it varies depth (skip layers) or tokens (route around blocks); essentially none of it varies how much of each weight matrix is materialized per token.

The gap at the intersection: a representation where the runtime decides, per layer and per token (or per short window), how many bytes of a nested weight encoding to read, with the rest resident only on SSD. Family 3 provides the encoding; family 4 provides the control policy; families 1–2 provide the redundancy evidence. No published system combines them (Phase 11 must re-verify this before any claim).

# 2. Techniques

# 2.1 SVD-family low-rank factorization (SVD, FWSVD, ASVD, SVD-LLM)

  • Mechanism. Replace W (m×n) with U_k Σ_k V_kᵀ (rank k), stored as two thin matrices. FWSVD weights the factorization by Fisher information of rows; ASVD scales by activation statistics; SVD-LLM adds truncation-aware data whitening (direct map from singular values to compression loss) plus a sequential parameter update after truncation.
  • Compression / bandwidth. Parameter ratio = k(m+n)/(mn). Bandwidth savings are proportional and dense — low-rank GEMMs are two ordinary GEMMs, no exotic kernels needed.
  • Quality (from papers). Vanilla SVD collapses quickly: at 40–60% compression, LLaMA-7B perplexities for SVD/FWSVD/ASVD reach the hundreds to tens of thousands. SVD-LLM keeps perplexity finite (e.g., ~13.1 at 40%, and 7.73 vs ASVD's 11.14 at 20% compression on LLaMA-7B), a >99% perplexity reduction vs prior SVD baselines — but average downstream accuracy still drops substantially at ≥40% (0.33–0.41 averages vs uncompressed ~0.6+). Consistent picture: weight matrices are not globally low-rank; only ~20–30% rank reduction is cheap.
  • Calibration. All modern variants need a small calibration set (whitening / activation stats); no retraining for SVD-LLM's base version.
  • Apple Silicon. Excellent: factors are plain dense matmuls, trivially expressible in MLX; can be combined with MLX affine quantization.
  • Limitation. Fixed rank chosen at compile time; quality cliff past moderate ratios; ignores that the residual W − U_kΣ_kV_kᵀ still contains most spectral energy in LLMs (singular values decay slowly).
  • Extension opportunity. Use the low-rank part as a resident hot path and keep the residual on SSD, loaded on demand — none of these papers store the residual at all.

# 2.2 Structural decomposition and rank-reduction as intervention (SliceGPT, LASER)

  • SliceGPT (ICLR 2024): applies orthogonal rotations (PCA of activations, exploiting computational invariance of pre-norm transformers) then deletes rows and columns, shrinking the embedding dimension. Removes up to 25% of parameters of Llama-2 70B / OPT-66B while keeping 99% zero-shot performance (90% for Phi-2); dense smaller matrices → real speedups (up to 1.55× throughput), no sparse kernels needed. Calibration only. Maps cleanly to MLX.
  • LASER (ICLR 2024): replacing selected weight matrices (mostly later-layer MLPs) with low-rank approximations improves task accuracy, at times by up to 30 percentage points, with no training. This is an intervention result, not a compression system, but it is first-order evidence that higher-order weight components of specific matrices are noise-like for some tasks.
  • Limitation / opportunity. Both are static. LASER's finding suggests per-matrix rank sensitivity is highly non-uniform — exactly the profile a compile-time analyzer (charter Phase 10 "compilation stage") should measure per layer.

# 2.3 Low-rank + quantized residual (CALDERA; randomized LPLR)

  • Mechanism. CALDERA (NeurIPS 2024): W ≈ Q + LR where Q is a full-rank but aggressively quantized backbone and L,R low-rank factors (also quantized, possibly at higher precision), obtained by calibration-aware alternating minimization with approximation-error bounds. Targets the 2–2.5 bit/param regime. Precursor: randomized low-rank + low-precision factorization (NeurIPS 2023) with explicit error bounds vs rank and bit-budget.
  • Relevance. This is literally the base + residual shape asked for in §4.5, with the roles inverted (quantized full-rank base + low-rank correction). The decomposition is additive, so the terms can in principle be loaded and applied independently — CALDERA never exploits that at runtime.
  • Apple Silicon. LR is dense GEMM (fine); Q uses QuIP#-style lattice codebooks in the reference implementation (CUDA kernels; Metal port nontrivial).
  • Limitation. Single fixed operating point; joint optimization couples the terms, so dropping one term is not quality-graceful by construction.
  • Extension. Re-derive the decomposition with a progressivity constraint (base alone must be usable) — cf. §2.8.

# 2.4 Codebook / vector quantization of weights (AQLM, QuIP#, GPTVQ, VPTQ)

  • Mechanism. Represent weight groups as sums of codebook entries: AQLM = additive multi-codebook quantization (from the retrieval literature, i.e. product/additive quantization à la Babenko) optimized against calibration activations; QuIP# = randomized Hadamard incoherence + E8 lattice codebooks + finetuning; GPTVQ, VPTQ = vector PTQ variants ("blessing of dimensionality").
  • Quality. Current SOTA at 2–3 bits/param. AQLM at ~4 bits: Llama-2 ppl 3.57 vs FP 3.46 (Wiki2, 70B-scale table in paper); AQLM repo reports a ~1-bit Llama-2-7B (1×8-bit codebook) at WikiText-2 ppl 7.85 (2025 update). QuIP# made 2-bit "viable" for the first time. Additive codebooks are inherently residual: each additional codebook refines the previous sum.
  • Calibration. Heavy (hours of optimization; QuIP#/AQLM benefit from finetuning).
  • Apple Silicon. Weakest point: efficient decode kernels are CUDA-only today; lattice decodes and LUT-heavy kernels need custom Metal work. MLX ships only affine (2/3/4/5/6/8-bit, group 32/64/128) and mxfp4/mxfp8/nvfp4 modes.
  • Limitation. Fixed bit-width at compile time; decoding cost nontrivial.
  • Extension. Drop-by-Drop / Matryoshka-supervised additive codebooks (§2.8) show the additive structure can be made ordered so codebooks can be dropped at runtime — the natural bridge to paging.

# 2.5 Cross-layer parameter sharing (ALBERT, Subformer, Relaxed Recursive Transformers, Basis Sharing, DeltaLLM, ResidualTransformer)

  • Mechanism. Reuse one block of weights across layers. ALBERT (all layers share), Subformer (sandwich sharing), Universal Transformer (recurrence) — train-from-scratch results. The post-training versions matter more here:
    • Relaxed Recursive Transformers (DeepMind, ICLR 2025): convert an existing LLM into a model that loops a small block of layers, "relaxed" by per-layer LoRA modules initialized via truncated SVD of the layer-vs-shared-weight difference. Recursive models converted from 2×-larger models can outperform same-size pretrained models; with distillation they approach the original.
    • Basis Sharing (ICLR 2025): express weights of different layers as combinations of a shared set of SVD-derived basis vectors + per-layer coefficients; outperforms SVD-LLM at 20–50% compression, calibration-only.
    • DeltaLLM (2025): share weights between adjacent transformer blocks and add low-rank per-layer deltas; ~30–40M tokens of light training; 12% parameter reduction retaining ~90% performance on Llama/Phi; DeltaPhi 2.9B (24% reduction) matches a finetuned SlicedPhi 3.3B. ResidualTransformer (ICASSP 2024) is the same idea for speech models.
  • Compression / bandwidth. This is the interesting part for us: a shared block resident in RAM amortizes across layers; per-layer deltas are small. A "one base layer + N cheap diffs" model turns per-layer weight traffic into per-layer delta traffic.
  • Limitation. Post-training conversion still needs some uptraining (RRT, DeltaLLM); pure zero-shot layer tying degrades badly. Compression ratios so far are modest (12–25%), far from the 10× regime.
  • Extension. Nobody treats the shared block as a cache-resident core and the deltas as SSD-resident pages. Also untested: sharing + progressive delta precision (delta rank/bits as a knob per layer).

# 2.6 Delta compression of finetunes (BitDelta, DeltaZip, Delta-CoMe)

  • Mechanism. Decompose a finetuned model as base + Δ and compress Δ: BitDelta quantizes Δ to 1 bit (sign + per-matrix scale, scales distilled in minutes); DeltaZip (EuroSys 2025) uses GPTQ-style compression of Δ (~10×) inside a multi-tenant serving system; Delta-CoMe (NeurIPS 2024) allocates mixed precision to Δ's singular vectors by singular-value magnitude (near-lossless at ~1-bit average, and unlike BitDelta it also holds up on math/code finetunes).
  • Quality. BitDelta: minimal degradation across Llama-2/Mistral/MPT up to 70B;

    10× memory reduction for multi-model serving. BitDelta's own ablation is notable for §4.6: applying BitDelta successively (compress, treat result as new base, compress the new delta…) yields an increasingly granular stack of 1-bit masks whose quality approaches the original monotonically — an accidental progressive code.

  • Relevance as evidence. Finetuning information ≈ 1 bit/param. This does not directly compress a base model, but it proves that "model = reference + extremely compressible correction" is a real structure in modern LLM weight space, and it motivates trying the same decomposition within one model (layer_i = layer_j + cheap delta; model = quantized self + cheap residual).
  • Apple Silicon. Sign matrices + scale are trivially Metal-friendly (1-bit masks decode to ±scale; MLX has no built-in kernel but the op is simple).

# 2.7 Kronecker and tensor-network decompositions (KnGPT2, TensorGPT, tensor trains)

  • Mechanism. W ≈ A ⊗ B (Kronecker; nearest-Kronecker via rank-1 SVD of reshaped W) or tensor-train factorization of reshaped weights/embeddings.
  • Quality. Results are only convincing at GPT-2/BERT scale with mandatory retraining (KnGPT2, ACL 2022). TensorGPT compresses embedding layers 2×–65× training-free on GPT-2-class models, but embeddings are a small fraction of a modern LLM. No competitive 7B+ results without heavy retraining found.
  • Verdict for us. Low priority: high implementation cost, weak post-training evidence at scale, and Kronecker-structured matmul kernels for Metal would be bespoke. Worth keeping only as a candidate basis for shared dictionaries.

# 2.8 Progressive / nested weight representations (Any-Precision LLM, MatQuant, BitStack, RRQ, Drop-by-Drop)

The family that matters most for localvm-research.

  • Any-Precision LLM (ICML 2024 oral): stores an n-bit (8-bit) "parent" model such that every k-bit child (3≤k<8) is obtained by taking the most significant bits — bit-plane overlay. Built post-training by "incremental upscaling" from a 3-bit seed (< 1 minute for 7B after seed quantization); ships a specialized (GPU) engine with bit-plane-aware memory layout. Memory: supporting {3..8}-bit Llama-2-7B costs 8.4 GB vs 29.9 GB for separate models (3.56×). Each bit-width matches SOTA quality for that width.
  • Matryoshka Quantization (MatQuant, DeepMind, ICLR 2025 oral): co-trains one int8 quantized model whose int4/int2 slices (MSBs) are all optimized jointly; int2 slices become up to ~10% more accurate than dedicated int2 QAT/OmniQuant — an int2-FFN Gemma-2 9B beats an int8-FFN Gemma-2 2B. Also allows layer-wise mix'n'match of precisions at inference. Requires QAT-style training.
  • BitStack (ICLR 2025): training-free. Iterative significance-weighted decomposition produces ~1-bit-per-parameter residual blocks; blocks are sorted (universally, across the whole model, by importance) and stacked in storage as transmission units; the runtime loads as many blocks as current memory allows → megabyte-level tradeoff between resident size and quality, matching or beating GPTQ/AWQ at extreme ratios. This is the closest existing system to "quality scales with bytes loaded." Known weakness (noted in follow-up work, e.g. the AMQ paper): on-the-fly weight reconstruction from residual blocks slows inference notably.
  • Recurrent Residual Quantization (RRQ, arXiv 2608.04048, 2026): calibration- free, additive stage-wise scheme — 2-bit RTN base + successive 2-bit RTN residual corrections gives 4/6/8-bit operating points from one package; the whole multi-precision package for Qwen3-8B builds in ~1,293 s (3.3× faster than MatGPTQ-style joint optimization).
  • Drop-by-Drop additive codebooks (arXiv 2606.12876, 2026): AQLM-style multi- codebook quantization with Matryoshka supervision so codebooks are ordered coarse→fine and can be dropped at inference for progressive compression.
  • Apple Silicon feasibility. Bit-plane and residual-stage layouts are exactly the kind of thing unified memory + mmap should be good at: each precision level is a separate contiguous region; upgrading precision = reading another region, not rewriting the resident one. No published Metal/MLX implementation of any of these exists (all engines are CUDA); MLX's affine quant kernels (2–8 bit) could serve stages if each stage is expressed as an affine-quantized tensor.
  • Common limitation. All of them select the operating point statically (per deployment, per memory budget). None selects precision per token; none ties the residual stages to storage paging; none reports bytes-read-per-token.

# 2.9 Early exit and dynamic depth (ACT, PonderNet, MSDNet, Depth-Adaptive Transformer, CALM, LayerSkip, Mixture-of-Depths)

  • Mechanism lineage. ACT (Graves 2016): learned halting for RNN steps. PonderNet (2021): stabilized probabilistic halting. MSDNet (ICLR 2018): anytime prediction with multi-scale features — the canonical "anytime NN". Depth- Adaptive Transformer (ICLR 2020): per-token decoder depth. CALM (NeurIPS 2022): confidence-gated early exit for LM generation with sequence-level calibrated guarantees (up to ~3× compute reduction, provably maintaining quality; addresses missing-KV problem of exited tokens). LayerSkip (Meta, 2024): layer dropout + shared early-exit head during training, then self-speculative decoding — early layers draft, remaining layers verify — 1.34–2.16× speedup with exact final quality. Mixture-of-Depths (2024): learned top-k token routing per block under a static compute budget.
  • Bandwidth reality check. Early exit saves depth — and therefore also the weight bytes of skipped layers for that token — but batch dynamics and KV bookkeeping erode the savings; and for us the key limit is that exit decisions gate whole layers, the coarsest possible granularity.
  • Retraining. CALM/LayerSkip/MoD all need training or finetuning with exit losses; nothing here is drop-in post-training on a frozen checkpoint (LayerSkip ships finetuned checkpoints; naive early exit on frozen models is poor).
  • Apple Silicon. Conceptually trivial to port (it is control flow, not kernels); single-request local decoding on a Mac is actually the friendly case (no batch synchronization problem).
  • Extension opportunity. LayerSkip's draft-then-verify structure is depth-based self-speculation. The unexplored dual: precision-based self-speculation — draft with a resident low-bit base (prefix of a nested representation, §2.8), verify/refine with residual planes only when the draft's top-1 margin is small. Verification reads extra bytes only on demand. (Charter Experiments D and G test exactly the preconditions.)

# 2.10 Approximate matrix multiplication (Drineas–Kannan–Mahoney, Bolt, MADDNESS)

  • Mechanism. (a) Randomized sampling: sample columns/rows with length-squared probabilities → unbiased estimate of AB with Frobenius error O(‖A‖‖B‖/√c) (DKM, SIAM J. Comput. 2006; foundation of RandNLA). (b) Learned LUT methods: Bolt, MADDNESS (ICML 2021) — replace one operand's inner products with learned hash-bucket lookups; up to 10× better speed-quality than prior AMM on small matrices, ~100× vs exact in the best cases.
  • Reality for LLMs. MADDNESS-class methods shine when one matrix is fixed and tall-thin regimes apply (classifier layers, kernels); accuracy at transformer scale is unproven, and LUT-gather-heavy inner loops are a poor match for GPU matmul pipelines (they beat CPUs, not tensor cores). Sampling-based AMM gives clean error bounds but errors are relative to matrix norms — too loose to certify token decisions directly.
  • Value to us. Not as a drop-in kernel, but as the theory toolbox for partial GEMM with error bars (Experiment E): length-squared/leverage sampling tells us which blocks matter most and what error skipping the rest costs — i.e., a principled block-ordering for progressive evaluation.

# 2.11 Adaptive-precision numerical computing (Wilkinson iterative refinement → GMRES-IR, five-precision IR)

  • Mechanism. Solve Ax=b with an LU factorization computed in low precision (cheap, fast), then iteratively refine: compute residual in high precision, solve a correction system (possibly by GMRES preconditioned with the low-precision factors), update. Carson & Higham (SIAM SISC 2018) formalized three-precision IR; Amestoy et al. extended to five precisions; NVIDIA/Dongarra demonstrated FP16-tensor-core factorizations refined to FP64 accuracy at ~4× speed.
  • Why it matters here. This is the canonical proof in numerical computing that "cheap approximate operator + residual-driven correction loop" recovers full accuracy while doing most work at low precision. The transformer analogue — run layers with a low-bit base, monitor a residual/confidence signal, apply stored higher-precision corrections only when needed — is structurally identical and appears untried for weights (speculative decoding is the analogue for tokens).
  • Caveat. IR has a convergence theory because A is the exact operator and the residual is exactly computable; in an LLM the "exact" layer output is not available without loading the full weights. The honest transferable idea is correction-on-demand plus a cheap instability detector (logit margins, §4.10 of the charter), not certified refinement.

# 2.12 Error-bounded lossy compression from HPC (ZFP, SZ)

  • Mechanism. ZFP (Lindstrom, TVCG 2014): fixed-rate or fixed-accuracy block transform coding of floating-point arrays, with published round-off error analysis (SIAM 2019) and random-access decode of 4^d blocks. SZ (Di & Cappello, IPDPS 2016): prediction + error-controlled quantization with strict pointwise error bounds; typically higher ratios than ZFP at equal bounds on many datasets.
  • Relevance. These are mature, error-budgeted, block-random-access codecs for float arrays — exactly the engineering shape a weight-paging store needs (bounded per-block reconstruction error → feeds §4.9 perturbation analysis; block random access → mmap-friendly pages). ZFP's fixed-rate mode gives predictable page sizes. Neither has been evaluated as an LLM weight format (weights are not smooth fields, so their predictors may underperform; needs Experiment-H-style measurement).
  • Apple Silicon. Both are C/C++ libraries that build on arm64; decode throughput vs Apple NVMe read speed is the number to measure.

# 3. Evidence of exploitable redundancy in pretrained transformers

The strongest measured facts found, ordered by how directly they support a base+residual execution model:

  1. Finetune deltas carry ≈1 bit/param of information. BitDelta quantizes the full delta of 7B–70B finetunes to 1 bit with minimal degradation; Delta-CoMe is near-lossless at ~1-bit average even for math/code finetunes. GPT-Zip/DeltaZip independently report ~10× delta compressibility. → Weight space has directions that are dramatically cheaper to encode relative to a reference.

  2. Adjacent layers are highly similar / near-linear. "Your Transformer is Secretly Linear" (ACL 2024) measures Procrustes linearity ≈0.99 between consecutive decoder layer embeddings across GPT/LLaMA/OPT/BLOOM, and shows some of the most-linear blocks can be removed or replaced by linear approximations with little loss. ShortGPT's Block Influence metric (cosine similarity between layer input and output) finds many layers barely transform the hidden state; removing them ("more redundant than you expect") costs little on benchmarks. "The Unreasonable Ineffectiveness of the Deeper Layers" (ICLR 2025) prunes large contiguous blocks of deep layers (selected by representational similarity) with minimal QA degradation after light QLoRA healing.

  3. Layer weights are compressible against each other. DeltaLLM: adjacent-block sharing + low-rank deltas retains ~90% performance at 12% reduction with only 30–40M tokens of training. Basis Sharing: one shared SVD basis serves multiple layers' weights with per-layer coefficients and beats per-layer SVD-LLM at 20–50% ratios. Relaxed Recursive Transformers: a looped shared block + SVD-initialized per-layer LoRA recovers most of the original model — i.e., much of a layer's identity is "shared trunk + small correction."

  4. Selective rank reduction can even help. LASER: replacing selected later-MLP matrices by low-rank approximations improves accuracy (up to +30 points on some tasks) — high-order components of specific matrices are noise-like. Complementary: SliceGPT removes 25% of parameters via activation-PCA rotation with 99% zero-shot retention on Llama-2 70B / OPT-66B.

  5. Task adaptation is intrinsically low-dimensional. Aghajanyan et al. (2020): RoBERTa-scale models can be finetuned to ~90% of full performance inside a random subspace of only ~hundreds of dimensions; pretraining reduces intrinsic dimension. → capability deltas, not just finetune deltas, are low-dimensional.

  6. But global low-rankness of weights is a myth. The SVD-family results (§2.1) consistently show steep quality loss past ~25–40% rank compression ("features are low-rank, weights are not"). Redundancy is structured (cross-layer, relative-to-reference, task-conditional) rather than uniform spectral decay.

  7. Inference is bandwidth-bound, so redundancy = latency. Every token reads every weight byte; on H100-class hardware compute outruns memory delivery by ~600× (Cloudflare "Unweight" engineering measurement). On Apple Silicon the ratio is smaller but the regime is the same — any byte not read per token is ~proportional latency, which is why decoupling "stored bytes" from "read bytes" (charter §2) is the right objective.

# 4. Progressive encodings from other fields

Transferable design patterns, from oldest to newest:

  • Embedded wavelet coding (EZW 1993, SPIHT 1996, JPEG2000/EBCOT). Coefficients are transmitted in significance order, bit-plane by bit-plane; the bitstream can be truncated at any byte and decodes to the best possible image for that byte count ("each new bit conveys the maximum information"). JPEG2000's EBCOT adds independently coded blocks with optimized truncation points — i.e., rate-distortion-optimal per-block truncation. Transfer: encode weight blocks as significance-ordered bit-planes/residual stages; "bytes loaded per matrix" becomes a continuous quality knob, and per-block truncation points can be optimized against layer sensitivity (Experiment F) instead of PSNR. BitStack (§2.8) is an unwitting rediscovery of this with residual SVD blocks; nobody has connected it to the mature R-D-optimal truncation machinery.

  • Progressive meshes (Hoppe, SIGGRAPH 1996) and Nanite (UE5). A mesh is stored as a coarse base + an ordered stream of refinements (vertex splits), giving lossless, continuous LoD, streaming, and selective refinement (refine only where the camera looks). Nanite industrializes this: fixed-size clusters in a hierarchical DAG, streamed on demand so only visible detail resides in memory, with LoD chosen per-cluster per-frame at ~pixel-error tolerance. Transfer: this is the exact architecture shape for weight paging — fixed-size weight "clusters" at multiple precisions, a residency set updated per token/ window by a cheap importance signal (attention/activation statistics as the "camera"), error tolerance expressed in logit margin instead of pixels.

  • Approximate query processing (Online Aggregation, SIGMOD 1997; BlinkDB, EuroSys 2013). Answer first, refine continuously, with statistical error bars; BlinkDB answers queries over 17 TB in <2 s within 2–10% error by choosing among precomputed stratified samples given a per-query time or error budget. Transfer: the interface idea — inference under an explicit (latency | error) budget, where the runtime chooses how much of the model to consult and can report confidence; and the offline idea — precompute multiple "samples" (precision profiles) of the model optimized for expected workloads.

  • Mixed-precision iterative refinement (Wilkinson 1963 → Carson–Higham 2018). See §2.11: do the O(n³) work in cheap precision once, recover accuracy with cheap corrective iterations. The pattern "expensive operator approximated + residual-driven correction + convergence monitor" is the numerical-analysis ancestor of any progressive-weight-refinement runtime.

  • Error-bounded scientific compression (ZFP/SZ). See §2.12: block random access + guaranteed per-element error bounds is the storage-format discipline a weight pager should adopt (bounded weight perturbation → bounded logit perturbation via layer Lipschitz estimates, rather than hoping).

# 5. Relevance to localvm-research

Could a progressive base+residual representation let quality scale with bytes loaded? The evidence says the ingredients all exist and individually work:

  • Nested/progressive weight codes exist and are near-SOTA at each operating point (Any-Precision LLM, MatQuant, BitStack, RRQ). BitStack already demonstrates monotone quality-vs-resident-megabytes on 7B–70B models, training-free.
  • Redundancy is real and structured (§3): a 2-bit-class base plausibly carries most behavior, and corrections are cheap relative to the base (BitDelta's iterated 1-bit masks converge to the original).
  • Control policies with quality guarantees exist for the depth dimension (CALM's calibrated exits; LayerSkip's exact self-speculative verification).
  • The systems patterns for demand-paged, error-budgeted, progressively refined data are mature in other fields (EBCOT truncation, Nanite residency, BlinkDB budgets, GMRES-IR refinement).

What has NOT been tried (candidate gaps for research_gaps.md):

  1. Token-/layer-conditional residual loading. Every progressive system picks its operating point statically per deployment. No published system decides per token (or per small window) and per layer how many residual stages to apply, despite Experiment-B/D-style predictability being the obvious enabler. The marriage BitStack × CALM does not exist.
  2. Precision-based self-speculation on a nested code. Draft with the resident low-bit prefix; verify/refine with SSD-resident residual planes only when the top-1 margin is small (LayerSkip's mechanism, transposed from depth to precision, with draft and verifier sharing the same bytes). Needs a Phase 11 novelty sweep (search terms: quantized self-speculation, precision cascade decoding, progressive dequantization inference).
  3. Bytes-read-per-token as the optimized objective. None of the §2.8 papers measures SSD/DRAM traffic per generated token; they measure resident size. On a 48 GB M5 Max with a fast NVMe, the interesting regime is: base resident (~2 bits/param), residual planes mmap'd, and a policy that keeps average bytes/token far below checkpoint size. This is measurable with our instrumentation plan (fs_usage, vm_stat) and no one has published it.
  4. R-D-optimal truncation for weights. Port EBCOT-style per-block optimized truncation to weight blocks, with distortion measured as calibration-set logit KL (not MSE), producing a layer-sensitivity-aware progressive layout at compile time (fits the charter's "compilation stage" exactly).
  5. Shared-basis trunk as the resident core. Basis Sharing / RRT / DeltaLLM suggest "shared trunk resident + per-layer deltas paged". Unexplored as a memory-hierarchy assignment rather than a compression ratio.

Apple Silicon specifics. Low-rank factors, sign-mask deltas, and affine- quantized stages (2–8 bit, group 32/64/128) map directly onto today's MLX kernels; bit-plane overlays and additive codebooks would need custom Metal kernels (all published engines are CUDA). Unified memory removes the CPU↔GPU copy that makes progressive loading painful on discrete GPUs: a residual plane read from NVMe into a mapped buffer is immediately GPU-visible. The BitStack-reported reconstruction slowdown is the main engineering risk — reconstruction must be fused into the matmul (dequant-in-kernel, as MLX already does for affine quant) rather than materialized. These claims about MLX/Metal feasibility are assessments to be validated in Experiments D/E/H, not established facts.

Failure modes to respect (charter §17): if per-token stage selection turns out to need near-all stages for acceptable quality (working set ≈ whole model), or if random 4–64 KB residual reads on Apple NVMe are too slow/thermally throttled (Experiment H), the progressive-paging premise dies; the fallback value of this literature is then "best static compressed format for MLX," which is already well-served by existing work.

# Sources