spb/localvm-research Public License

1---2project: localvm-research3document: research/notes/quantization4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6created: 2026-08-117status: draft8---910# Quantization — deep literature note (charter §4.1)1112Scope: post-training quantization (PTQ), quantization-aware training (QAT), 8→1-bit and ternary regimes, mixed/per-layer/per-channel/dynamic/progressive precision, residual/additive/vector/lattice/trellis (codebook) quantization, weight-only vs weight+activation quantization, KV-cache quantization, and extreme low-bit inference — with explicit attention to (a) **memory-capacity reduction vs memory-bandwidth reduction as distinct effects**, and (b) **which formats have efficient Metal/MLX kernels vs CUDA-only implementations**. All sources listed in §Sources were accessed 2026-08-11.1314---1516## 1. Landscape overview1718The field has converged on a fairly stable picture as of mid-2026:19201. **4-bit weight-only is a solved commodity.** GPTQ (ICLR 2023), AWQ (MLSys 2024 best paper), and calibration-free HQQ all produce 4-bit models within a few percent (often <0.1–0.3 PPL) of FP16. Every serious Mac runtime (llama.cpp Metal, MLX) executes 4-bit weights natively at close to memory-bandwidth limits.212. **2–3-bit is where the frontier is.** State of the art moved from scalar quantization (GPTQ) → incoherence processing + scalar (QuIP) → vector/lattice codebooks (QuIP#, AQLM, GPTVQ) → trellis-coded quantization (QTIP, EXL3), each step buying quality at the same bitrate. QuIP# was the first PTQ method where 3-bit scaled better than 4-bit. The catch for us: **all SOTA sub-3-bit codebook/trellis kernels are CUDA-first**; on Apple GPUs the analogous formats (llama.cpp i-quants) decode measurably slower than simple affine formats.223. **Weight+activation quantization (W8A8, W4A4) is a compute/throughput play, not primarily a capacity play** — relevant for prefill speed, mostly irrelevant to the decode-time bandwidth wall on a Mac at batch 1. Rotation methods (QuaRot, SpinQuant) made W4A4 viable by killing activation outliers.234. **Sub-2-bit exists but changes character.** BiLLM (PTQ, ~1.08 bit) is "viable" but degraded; BitNet-style ternary needs (re)training; ParetoQ shows a sharp representational transition between 2 and 3 bits — below it, weights leave the pretrained basin. For a system that must preserve the original model's behavior (charter §14 hard constraint), 2-bit is the practical floor for a *base* representation, not an endpoint.245. **The newest and, for us, most important thread is multi-precision / nested / progressive representations**: Any-Precision LLM (bitplanes, ICML 2024 oral), Matryoshka Quantization (MSB-nesting, ICLR 2025 oral), Drop-by-Drop additive codebooks with successive-refinement theory (2026), and Progressive Mixed-Precision Decoding (ICLR 2025). These decouple *stored* bits from *read* bits — exactly the `total size ≠ bytes read per token` decoupling in our core research question. None of them targets Apple Silicon, and none streams residual precision from SSD on demand.2526Key systems distinction used throughout: at batch-1 decode, token latency ≈ (bytes of weights+KV read per token) / (achievable memory bandwidth). A format reduces **memory** if the resident representation is smaller; it reduces **bandwidth per token** only if the bytes actually *traversed* per token shrink and the dequant compute doesn't become the new bottleneck. These usually coincide for dense inference (every weight is read every token) but *diverge* for: codebook lookups (extra random reads), lossless coding (decode compute), bitplane/nested formats (read fewer planes than stored), and any offload scheme.2728---2930## 2. Technique families3132### 2.1 Round-to-nearest and second-order weight-only PTQ (GPTQ family)3334**What it does.** GPTQ ("Accurate Post-Training Quantization for Generative Pre-trained Transformers", Frantar et al., ICLR 2023) quantizes weights column-by-column, using a per-layer Hessian proxy (from ~128 calibration samples) to update remaining unquantized weights and cancel accumulated error. Quantized OPT-175B to 3–4 bits in ~4 GPU-hours.3536- **Memory reduction:** ~4× at 4-bit, ~5.3× at 3-bit vs FP16.37- **Bandwidth reduction:** proportional to memory for dense decode (all weights read each token); reported 3.25× (A100) / 4.5× (A6000) generation speedups vs FP16.38- **Quality:** near-lossless at 4-bit (e.g., OPT-1.3B: FP16 PPL 14.63; naive RTN 4-bit 48.24; GPTQ 4-bit 15.47). 3-bit usable; 2-bit/ternary "viable" only with tiny groups. AWQ's authors document GPTQ overfitting to the calibration distribution (2.3–4.9 PPL worse under calibration/eval distribution shift, vs 0.5–0.6 for AWQ).39- **Calibration/retraining:** calibration data required; no retraining.40- **Apple Silicon status:** the *algorithm* is platform-neutral; GPTQ-produced integer weights convert to GGUF/MLX layouts fine. The fast GPTQ inference kernels (Marlin, ExLlama) are CUDA-only. mlx-lm ships a GPTQ-style learned-quant recipe natively.41- **Main limitation:** scalar, uniform grids; hits a wall below 3 bits.42- **Extension opportunity for us:** GPTQ's Hessian machinery is reusable for *allocating* precision (which blocks deserve residuals) rather than just rounding.4344**AWQ** (Lin et al., MLSys 2024 Best Paper, arXiv 2306.00978): observation that ~1% of weight channels are salient *as measured by activation magnitude*; protecting them via per-channel scaling (no mixed precision needed) preserves quality. No backprop; more robust to calibration shift than GPTQ. TinyChat gives >3× over HF FP16 on desktop/mobile GPUs. mlx-lm includes an AWQ-style learned quant. Same memory/bandwidth profile as GPTQ.4546**HQQ** (Mobius Labs, blog Nov 2023): calibration-**free** — solves a per-group half-quadratic optimization on weights only. Llama-2-70B quantized in <5 min (>50× faster than GPTQ); their 2-bit 70B beats FP16 Llama-2-13B in PPL at comparable memory. Useful to us as the cheapest way to generate base representations at many bit-widths for experiments. Python/PyTorch; runs anywhere (incl. MPS backend) but fast fused kernels are CUDA.4748**SqueezeLLM** (Kim et al., ICML 2024, arXiv 2306.07629): sensitivity-based **non-uniform** (k-means) codebooks per output channel + **dense-and-sparse decomposition** (0.05–0.45% of outlier/sensitive weights kept in FP16 sparse format). 3-bit LLaMA-7B: sensitivity-weighted clustering takes PPL from 18.08 (unweighted k-means) to 7.75. Explicitly frames single-batch LLM inference as **memory-bandwidth bound**, and shows LUT dequant can still hit ~2.3× GPU speedup. The dense+sparse split is a primitive "base + residual" — the sparse part is a tiny, importance-selected correction stream. CUDA kernels only.4950### 2.2 Weight + activation quantization (SmoothQuant → rotations)5152**SmoothQuant** (Xiao et al., ICML 2023, arXiv 2211.10438): migrates activation-outlier difficulty into weights via per-channel equivalent scaling → W8A8 training-free, negligible loss on OPT/BLOOM/Llama; up to 1.56× speedup and 2× memory reduction; enabled serving 530B in one node.5354**QuaRot** (Ashkboos et al., NeurIPS 2024): computational-invariance rotations (randomized Hadamard) applied to hidden state, FFN activations, attention, and KV cache remove outliers *without changing model output*, enabling **end-to-end 4-bit** (weights, activations, KV). Llama-2-70B W4A4: ≤0.47 WikiText-2 PPL loss, 99% zero-shot retention; lossless W6/W8 with plain RTN and *no calibration data*.5556**SpinQuant** (Liu et al., ICLR 2025): same invariance idea but with *learned* rotations (Cayley-optimized R1/R2 absorbed offline, plus online Hadamard R3/R4), closing more of the W4A4KV4 gap than random rotations.5758- **Memory/bandwidth:** activation quantization does not shrink the checkpoint; its win is compute density (INT8/INT4 matmul) and KV size. On a Mac at batch 1 decode this matters mainly through the KV cache and prefill speed.59- **Apple Silicon status:** no shipped Metal W4A4 path. MLX recently added `quantize_input` and mxfp8/nvfp4 modes (see §2.4), which is the beginning of an activation-quantization story; fast Hadamard transform kernels for Metal would need to be written. Note M5-class Neural Accelerators change the compute/bandwidth ratio in favor of such schemes.60- **Relevance:** incoherence/rotation preprocessing is *upstream-compatible* with any base representation we choose — it makes weights more Gaussian, which is exactly what codebook, lattice, and progressive-residual encodings want. QuIP#/QTIP already rely on it.6162### 2.3 GGUF block formats: k-quants, i-quants, ternary types (llama.cpp)6364**What they are.** llama.cpp's zoo: legacy Q4_0/Q4_1/Q5_x/Q8_0 (per-block scale ± min); **k-quants** (Q2_K…Q6_K) with two-level scale hierarchies and per-tensor mixes (the `_S/_M/_L` file types give sensitive tensors more bits); **i-quants** (IQ1_S…IQ4_XS) which are codebook/grid-based sub-4-bit formats requiring an **importance matrix** (imatrix, calibration-derived) for quality; ternary **TQ1_0/TQ2_0**; and **MXFP4** (added Aug 2025 for gpt-oss, whose FFN weights ship natively in MXFP4).6566- **Memory/bandwidth:** proportional (dense decode). Q4_K_M ≈ 4.9 GB for an 8B model. Practical measurements put Q4_K_M ~+0.08 PPL over FP16 on Llama-3-8B class models.67- **Quality per bit:** k-quants beat MLX affine at matched bpw — one practitioner measurement (Feldman, 2026): Q4_K_M at 4.88 bpw gives 0.0208 nats KL-to-FP16 vs MLX affine 4-bit (4.69 bpw) 0.0577 nats, ~2.8× better; similar at 5-bit. The reason is the two-level scale hierarchy and mixed per-tensor precision.68- **Apple Silicon / Metal status — the critical bandwidth-vs-compute datapoint:** all GGUF types have hand-written Metal kernels, but **i-quants pay a large decode penalty on Apple GPUs**. ikawrakow (llama.cpp discussion #5617): 7B IQ-quant ~53.9 t/s vs Q4_0 63.1 t/s on M2 Max 30-core GPU, despite reading ~half the bytes; vs an RTX-4080 the gap for IQ2_XS is 3.5× (only 2× for Q4_0). Community measurements agree the codebook lookups make dequant compute-bound on Apple Silicon (and older CPUs). **Lesson: on Metal, "fewer bits" only converts to "faster tokens" if the decode path stays trivially cheap (shift/mask + multiply-add), which is exactly what k-quants and MLX affine do and what LUT-heavy formats do not.**69- **Limitation:** static, uniform-per-tensor decisions; no notion of loading *part* of a weight's information.70- **Extension:** the imatrix (importance) infrastructure is a ready-made per-block sensitivity signal usable for residual allocation.7172### 2.4 MLX native quantization (our primary substrate)7374**Formats** (mlx.core.quantize / mlx.nn.quantize docs, MLX 0.32): default **affine** mode — bits ∈ {2,3,4,5,6,8}, group size ∈ {32,64,128} (default 4-bit/g64), per-group scale+bias, ŵ = round((w−β)/s); plus **mxfp4** (E2M1, g32, E8M0 scale), **mxfp8** (g32), **nvfp4** (g16, E4M3 scale). `quantize_input=True` enables input/activation quantization for linear layers. All modes have first-class Metal kernels (fused dequant-matmul); quantized-KV attention supported via `mlx_lm.generate --kv-bits {2,4,8} --kv-group-size`.7576**Learned quants in mlx-lm** (LEARNED_QUANTS.md): **DWQ** (distilled weight quantization — distills a 16/8-bit teacher into the *quantization parameters* (scales/biases) of a low-bit student; works best at 2–4 bit; community measurements ≈ +0.6 effective bits of quality); **AWQ** port; **GPTQ**-style; **dynamic_quant** (per-layer sensitivity-based bit allocation: layers that hurt most get more bits). This is the closest thing to a maintained, Apple-first learned-PTQ toolchain, and it's pure Python on top of MLX — directly hackable for our experiments.7778- **Memory/bandwidth:** proportional; MLX 4-bit ≈ 4.5 GB for 8B. MLX vs llama.cpp decode speed on identical Macs is within ±10–20% either way depending on version/model; MLX tends to win on 4-bit 7–30B decode, llama.cpp on prefill (mature simdgroup matmuls). Ollama switched its Apple Silicon backend to MLX (2026), and M5 "Neural Accelerators" give MLX further headroom.79- **Quality:** plain affine 4-bit/g64 trails Q4_K_M slightly (see §2.3); group-size 32 and/or DWQ closes the gap.80- **Main limitation:** uniform affine only — no non-uniform codebooks, no two-level scales, no sub-2-bit, no nested/progressive layout. Quantization is chosen once at convert time; the runtime has no concept of refining a weight after load.81- **Extension:** MLX's `mode` plug-point plus custom Metal kernels is where a progressive/residual format would be implemented. The DWQ distillation loop is also the obvious way to *calibrate a low-bit base to be maximally correctable by its residuals*.8283### 2.5 Vector / additive / lattice / trellis codebook quantization (the 2-bit frontier)8485**QuIP** (Chee et al., NeurIPS 2023): incoherence processing (random orthogonal pre/post rotations) + adaptive rounding; first theoretical analysis at LLM scale; first "viable" 2-bit.8687**QuIP#** (Tseng et al., ICML 2024): randomized Hadamard incoherence + **E8-lattice 8-dimensional codebook (E8P)** + inter-layer fine-tuning. Higher bitrates built via **residual vector quantization** (RVQ — quantize, then quantize the residual with another codebook; e.g., 4-bit = 2+2). Llama-2-70B Wiki2 PPL (no FT): FP16 3.12 → E8P 2-bit 4.16 (vs 5.90 for QuIP scalar 2-bit). First PTQ where 3-bit scaled better than 4-bit. >3× faster than FP16 inference (CUDA); codebook decodable in <4 instructions/weight due to E8 symmetry.8889**AQLM** (Egiazarian et al., ICML 2024, arXiv 2401.06118): **additive quantization** — each weight group is a *sum of several codewords* from learned 8-dimensional, 2^16-entry codebooks, trained by beam search + block-wise then end-to-end fine-tuning. First scheme Pareto-optimal below 3 bits; Pareto-optimal bitwidth ≈ 2.5 bpw. **PV-Tuning** (NeurIPS 2024) adds representation-agnostic fine-tuning of discrete+continuous params → first Pareto-optimal 2-bit Llama-2. Cost: ~720 GPU-hours for a 70B; 1 MiB codebooks blow L1 cache → AQLM decode is *slow* (20.6 tok/s vs QuIP# 106.3 on 2-7B, per QTIP measurements).9091**GPTVQ** (van Baalen et al., Qualcomm, ICML 2024): GPTQ-style Hessian-interleaved updates extended to non-uniform VQ (1–4D); codebooks themselves compressed (int + SVD). 70B processed in 3–11 h. Notably demonstrated **simultaneous DRAM-footprint and latency reduction on a mobile-class Arm CPU** — one of the few codebook methods validated on unified-memory consumer silicon rather than discrete GPUs.9293**QTIP** (Tseng et al., NeurIPS 2024): **trellis-coded quantization** — stateful codes over long sequences instead of fixed-dim VQ; with compute-based "bitshift trellis" codebooks there is no large LUT, decode is ~2 instructions/weight, and matvecs run at >80% of peak GPU memory bandwidth. Beats QuIP#/AQLM at all bitrates; QTIP-without-finetune ≈ QuIP#/AQLM-with-finetune. **EXL3** (exllamav3) is a streamlined QTIP variant productized for consumer GPUs — Llama-3.1-70B coherent at 1.6 bpw, 70B in <16 GB VRAM. **CUDA-only.**9495**NestQuant** (Savkin et al., ICML 2025, arXiv 2502.09720): self-similar **nested lattices** (Gosset/E8), information-theoretically near-optimal for low-precision matmul, covering weights *and* activations. Also NeurIPS 2025 work on learned grouped lattice VQ (learnable generator matrices, Babai rounding). Lattice methods are converging with VQ methods.9697- **Memory reduction:** the whole point — 2.0–2.5 bpw with usable quality (~8× vs FP16).98- **Bandwidth reduction:** *conditional*. Weight bytes drop 8×, but decode cost decides whether that becomes tokens/sec: QTIP/QuIP# reach near-bandwidth-limit on NVIDIA; AQLM does not; and the Apple-GPU evidence from i-quants (§2.3) says LUT-based decode may leave Metal compute-bound. No published Metal implementation of QuIP#/AQLM/QTIP exists as of this writing.99- **Calibration:** all need calibration; AQLM/QuIP#/QTIP quality depends significantly on (expensive) fine-tuning.100- **Main limitation for us:** CUDA-first ecosystems; heavy encode cost; fixed bitrate at encode time.101- **Extension opportunity:** **RVQ/additive structure is inherently progressive** — codeword sums can be truncated (see Drop-by-Drop, §2.7). A Metal "bitshift-trellis" decoder (no LUT) is plausibly the right way to get QTIP-class quality on Apple GPUs; nobody has published one.102103### 2.6 Extreme low-bit: ternary, 1-bit, and low-bit QAT104105**BitNet b1.58** (Ma et al., Microsoft, arXiv 2402.17764): ternary {−1,0,+1} weights, *trained from scratch*; claims parity with FP16 at same params/tokens, with large latency/memory/energy wins. Not a post-training transform → excluded as a solution by charter §14, but relevant as an existence proof of ~1.58-bit information sufficiency and for its inference stack. **bitnet.cpp** (arXiv 2410.16144; ACL 2025): CPU LUT kernels — I2_S (lossless MAD-based), TL1 (ARM), TL2 (x86); 1.37–5.07× speedups on ARM; runs a demo on Apple M2, and TL2_0 reaches 7.45 tok/s for a **100B ternary model on an M2 Ultra** — an interesting datapoint that sub-2-bit makes 100B-class models CPU-feasible on Macs. Note: ternary kernels are **CPU-only** (NEON/AVX LUTs), no Metal GPU path; llama.cpp's TQ1_0/TQ2_0 exist but were measured badly inaccurate for BitNet models by the bitnet.cpp authors.106107**BiLLM** (Huang et al., ICML 2024): first 1-bit PTQ — Hessian-selected salient weights get **binary residual approximation** (a second binary pass over the residual: again base+residual!), non-salient bell-shaped weights get optimal-split binarization. ~1.08–1.11 bpw; LLaMA2-70B PPL 8.41; 7B binarized in 0.5 h on one GPU. Quality is far from FP16 (usable-ish, clearly degraded); ARB-LLM (2024) refines it.108109**ParetoQ** (Liu et al., Meta, NeurIPS 2025, arXiv 2502.02631): unified QAT-finetune framework across 1/1.58/2/3/4-bit. Key findings: ~10% of training budget on QAT finetuning suffices; **sharp representational transition between 2 and 3 bits** — at ≥3 bits finetuned models stay close to the pretrained distribution ("compensation"); at ≤2 bits representations change drastically ("reconstruction"). Ternary/2-bit/3-bit beat 4-bit on size-accuracy Pareto in their runs.110111**EfficientQAT** (Chen et al., ACL 2025, arXiv 2407.11062): block-wise QAT (Block-AP) + end-to-end training of quant params only (E2E-QP). 2-bit Llama-2-70B on a single A100 in 41 h, −3% accuracy vs FP16 (69.48 vs 72.41); w2g64 Llama-2-7B PPL 6.86 vs 5.47 FP16. This is the realistic quality ceiling for "2-bit that still behaves like the original model" with modest compute.112113- **Implication for localvm-research:** ParetoQ's 2↔3-bit transition + EfficientQAT numbers suggest a **2-bit base is recoverable with light training but is near the edge**; a 2.5–3-bit-effective base (or 2-bit + streamed residual) is the safer floor if the base must preserve routing/decision structure without full QAT.114115### 2.7 Multi-precision, nested, progressive, and residual representations ⭐ (most relevant family)116117**Any-Precision LLM** (Park et al., ICML 2024 **oral**, arXiv 2402.10517; code SNU-ARC/any-precision-llm): store ONE n-bit "parent" model (non-uniform, incremental-upscaling from a 3-bit seed) in **bitplane layout**; any child bit-width 3…n is obtained by reading only the top-k bitplanes. Memory: supporting {3,4,5,6,7,8} bits costs 8.4 GB instead of 29.9 GB for separate models (3.56× saving). Crucially, **"any runtime request of reduced bit-width directly translates into proportional speedup, as we can simply load the specified number of bits"** — i.e., bytes-read-per-token scales with *chosen* precision, not stored precision. Engine has custom **CUDA** kernels (bitplane layout, bit-transpose, merged table lookups). No Metal port exists.118119**Matryoshka Quantization / MatQuant** (Nair et al., Google DeepMind, ICLR 2025 oral, arXiv 2502.06786): exploit MSB-nesting of integers — co-optimize (via QAT or OmniQuant-style PTQ) a single int8 weight tensor so that slicing its top 4 or 2 bits yields good int4/int2 models; int2 extracted this way is up to 10% more accurate than dedicated int2 QAT; an int2-FFN Gemma-2 9B beats an int8-FFN Gemma-2 2B. Interpolative bit-widths (int3/int6) come for free.120121**Drop-by-Drop / multi-bitwidth additive codebooks** (arXiv 2606.12876, Feb 2026): grounds multi-precision PTQ in **information-theoretic successive refinement**; trains additive (AQLM-style) codebooks with Matryoshka supervision so that *ordered subsets of codebooks* give accurate partial reconstructions — progressive compression by literally dropping codebooks at inference. Explicitly motivates "run-time hardware-aware dynamic loading". This is the closest published object to a "progressive residual base representation," and it is brand new — no systems implementation, no SSD tier, no Apple port.122123**Progressive Mixed-Precision Decoding (PMPD)** (Chen et al., Samsung AI, ICLR 2025, arXiv 2410.13461): *phase-aware* precision — higher-precision weights for prefill, lower for decode, and **progressively lower precision as generation deepens** (later tokens tolerate more error), with task-/prompt-adaptive schedulers. 3.8–8.0× decode throughput on an NPU. Validates, at small scale, the premise that required weight precision is *token-position-dependent* and can be scheduled at runtime.124125Related: **AnyBCQ** (2025) — binary-coded multi-precision with hardware-efficient bit-plane access; **Squeeze10-LLM** (2025) staged sub-2-bit PTQ; the **mixed-precision survey** arXiv 2510.16805 (Oct 2025) taxonomizes this whole space (§3.1 covers Any-Precision, PMPD, M2Cache).126127- **Memory:** one artifact serves all precisions (≈ cost of the highest).128- **Bandwidth:** the *defining* feature — bytes/token = f(precision requested now), decoupled from storage. This is breakthrough criterion B in embryo.129- **Quality:** MatQuant int2-slice ≈ or better than dedicated int2; Any-Precision children match dedicated SqueezeLLM-class models at each width.130- **Apple status:** none. All engines are CUDA (or NPU simulators). Bitplane decode is bit-manipulation-heavy — Apple GPU/AMX behavior unknown; a Metal bitplane-matvec kernel is an obvious micro-experiment (ties to charter Exp. D/E).131- **Limitation:** all of these still *load the full chosen precision for every weight every token* — precision is scheduled globally (per phase/token), not per weight-block; and the lowest usable slice is ~2–3 bits.132133### 2.8 Random-basis and lossless recompression (orthogonal tricks)134135**SeedLM** (Shafipour et al., **Apple** + Meta, ICLR 2025, arXiv 2410.10714): compress each weight block into a **seed + coefficients of an LFSR-generated pseudo-random basis** — at inference, regenerate the basis from the seed and reconstruct the block. Data-free, 3–4-bit effective, ~zero-shot parity with calibration methods at 4-bit; trades **memory bandwidth for free compute** (PRNG regeneration), which is exactly the right trade on bandwidth-bound hardware — FPGA demo ~4× speedup at 70B. Apple authored this: conceptually adjacent to "weights need not be stored, only recoverable."136137**DFloat11** (Zhang et al., NeurIPS 2025, arXiv 2504.11651): **lossless** Huffman coding of BF16 exponents → ~30% size cut (≈11 bpw), bit-identical outputs, GPU online-decompression kernels. Cautionary datapoint: even with careful kernels, decode costs ~2–3× tokens/s vs uncompressed-in-VRAM on 8–32B models (it only wins when the alternative is offload). **Lossless coding shrinks capacity, not effective bandwidth** — entropy decode sits on the critical path. CUDA-only.138139### 2.9 KV-cache quantization140141- **KIVI** (Liu et al., ICML 2024, arXiv 2402.02750): tuning-free 2-bit KV — **keys per-channel, values per-token** (asymmetric, matches outlier structure), small FP16 sliding window; 2.6× peak-memory cut, 2.35–3.47× throughput (batch effect). Inspired HF Transformers' KV quantization.142- **KVQuant** (Hooper et al., NeurIPS 2024, arXiv 2401.18079): per-channel pre-RoPE key quant + non-uniform sensitivity-weighted datatypes + per-vector dense-and-sparse outliers → **3-bit KV with <0.1 PPL degradation**; enables 1M-token context for LLaMA-7B on one A100.143- **Coupled Quantization** (NeurIPS 2024): exploits inter-channel dependence to reach ~1 bit/channel-equivalent KV.144- **Apple Silicon status — good:** llama.cpp `--cache-type-k/v {q8_0,q5_0,q4_0,iq4_nl}` with flash attention on Metal (q8_0 ≈ half KV memory, negligible loss; q4_0 noticeable on long reasoning); mlx-lm `--kv-bits {2,4,8}` (group 64 default). Community work (KVSplit) confirms asymmetric K/V precision pays on Metal. Google's 2026 sub-3-bit KV result (picked up in llama.cpp discussion #20969, "TurboQuant", with working Metal kernels at 3.25/4.25 bits) is being adopted.145- **Relevance:** KV bytes/token grow with context and can rival weight bytes at long context on 48 GB; any working-set argument must model both. KV quantization is the *already-solved* part of the working-set problem on Macs.146147### 2.10 Precision granularity: per-layer / per-channel / per-token / dynamic148149Cross-cutting rather than a single method: per-channel scales are standard (AWQ/GPTQ groups); per-layer bit allocation is in mlx-lm `dynamic_quant` and GGUF `_M` mixes; per-token/phase precision is PMPD (§2.7); dynamic runtime precision selection appears in Any-Precision serving and M2Cache (below). The mixed-precision survey (arXiv 2510.16805) is the map of this space. **MoE note:** for MoE models, per-expert precision (hot experts high-bit in RAM, cold experts low-bit near SSD) is an obvious composite nobody ships on Mac; but the energy study arXiv 2508.06978 warns that *naive* SSD expert-offload raises per-token energy up to ~12× vs HBM — prefetch hides latency, not energy or bandwidth.150151### 2.11 Quantization + offload hybrids (closest existing systems to our target)152153**M2Cache** (arXiv 2410.14740): neuron-level modularization + importance ranking + **dynamic sparse mixed-precision quantization** + three-level cache (GPU HBM ← DRAM ← SSD). Highest-importance neurons stay high-precision and cached; low-importance ones live at low precision on lower tiers. Up to 14× throughput vs baseline offload on RTX 3090-class hardware, 70B on 24 GB VRAM. This is the closest published architecture to charter §13 — but: Linux/CUDA, discrete-GPU tiering assumptions (PCIe copy costs that don't exist on unified memory), static importance ranking, and no progressive-precision refinement (a neuron is fetched at one precision, not refined).154155---156157## 3. Relevance to localvm-research158159### 3.1 What quantization gives us as a base representation160161The working-set-decoupling question needs a representation where `bytes read per token` is a *runtime variable*. The literature supplies four composable building blocks:1621631. **A 2–4-bit affine base with native Metal speed** (MLX affine / k-quants) — the only formats today where fewer bits reliably equal proportionally fewer nanoseconds on Apple GPUs. A 2-bit/g32 (+DWQ-calibrated) MLX base is implementable *now*; expected quality per EfficientQAT/ParetoQ: degraded but structurally faithful (PPL +~1.4 on 7B-class at w2g64 with training; worse pure-PTQ).1642. **Nested/bitplane layouts** (Any-Precision, MatQuant) — store 6–8 bits, *read* 2–4. MSB-slicing means the "residual" is literally the next bitplane: refinement = read more planes of the same tensor, perfectly sequential, prefetchable, and idempotent. This is the natural on-SSD layout for progressive weight materialization (charter Exp. D).1653. **Additive/RVQ codebooks with successive-refinement training** (QuIP#'s RVQ, Drop-by-Drop) — higher quality per bit than bitplanes at the same budget, refinement = add codewords; but decode-cost risk on Metal (i-quant lesson) unless a QTIP-style computed (LUT-free) codebook is used.1664. **Error-side instruments**: SqueezeLLM/BiLLM's dense-and-sparse and KVQuant's outlier streams show that a *tiny, importance-ranked sparse residual* captures outsized quality — a cheap, cache-friendly correction channel that can be paged independently of the dense base.167168### 3.2 The specific opening (what is NOT in the literature)169170Verified against everything above, the following combination does not exist:171172- **Progressive precision as a memory hierarchy.** Any-Precision/MatQuant/Drop-by-Drop keep all bitplanes/codebooks *resident* and slice for latency; PMPD schedules precision *globally per token phase*. **Nobody stores the low-bit base resident in unified memory and treats higher-order residual planes as a demand-paged tier on NVMe**, fetched per-block, conditioned on (layer sensitivity × current-token need × cache state). That system would make RAM bound the *base* size (e.g., 2 bits/param ≈ 17.5 GB for a 70B) while total quality lives on disk — precisely `resident size ≠ total size ≠ bytes/token`.173- **Decision-stability-driven refinement.** No published method decides *how many residual planes to fetch* based on whether the token decision (top-1 margin / top-k set) is already stable — PMPD's schedulers are position-based, not confidence-based; Drop-by-Drop's profiles are static. This connects quantization directly to charter §4.10 / Exp. G: fetch residuals only when the low-bit forward pass is uncertain.174- **Apple Silicon kernels for any of it.** No Metal implementations exist for: bitplane matvec (Any-Precision), MSB-sliced decode (MatQuant), trellis decode (QTIP/EXL3), E8/additive codebooks (QuIP#/AQLM). Unified memory actually *helps* here versus CUDA: a refined block is written once and visible to the GPU without PCIe traffic, and the SSD→RAM→GPU path has no copy step. The i-quant Metal evidence defines the design constraint: decode must be shift/mask-cheap (bitplanes and bitshift-trellises qualify; big LUTs do not).175- **Residual-aware calibration.** DWQ/EfficientQAT optimize a single bit-width; MatQuant co-optimizes slices; but nobody calibrates a base *jointly with a sparse importance-ranked residual stream under a bytes-per-token budget* ("make the 2-bit base maximally correctable by its cheapest residuals"). mlx-lm's DWQ loop is the natural place to prototype this on-device.176177### 3.3 Falsifiable premises to test first (feeds Exp. D/E/H)1781791. **Metal bitplane decode cost:** does a 2-of-8-bitplane matvec on M-series GPU run at ≥70% of the speed of a dense 2-bit affine matvec? (If not, MSB-sliced *packed* MatQuant layout instead of bitplanes.)1802. **Quality-vs-planes curve:** on a 7–8B model, measure PPL/KL/greedy-token-agreement at 2, 2+1, 2+2 … planes (charter Exp. D). MatQuant/Any-Precision predict smooth improvement; the open question is how *few blocks* need refinement to recover most of it.1813. **Residual locality:** are the blocks whose refinement matters stable across tokens/domains (Exp. B/C)? If yes, an SSD residual tier with an LRU of refined blocks beats static mixed precision; if no, bandwidth math likely kills it (cf. arXiv 2508.06978's energy warning).1824. **Bandwidth accounting:** a 48 GB M5 Max has O(500+) GB/s unified memory vs O(6–8) GB/s NVMe — residual fetches must therefore be ≤~1–2% of weight bytes per token, or fully overlapped/amortized across tokens. This ratio, not quality, is the most likely failure mode; measure before building (Exp. H).183184---185186## Sources187188- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (Frantar et al., ICLR 2023) — https://arxiv.org/abs/2210.17323 (accessed 2026-08-11)189- AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (Lin et al., MLSys 2024) — https://arxiv.org/abs/2306.00978 (accessed 2026-08-11)190- SmoothQuant: Accurate and Efficient Post-Training Quantization for LLMs (Xiao et al., ICML 2023) — https://arxiv.org/abs/2211.10438 (accessed 2026-08-11)191- A Practical Guide to INT4 Quantization for SLMs: GPTQ vs AWQ (Microsoft Data Science, Medium) — https://medium.com/data-science-at-microsoft/a-practical-guide-to-int4-quantization-for-slms-gptq-vs-awq-olive-and-real-world-results-2f63d6963d1d (accessed 2026-08-11)192- Combining multiple post-training techniques to achieve most efficient quantized LLMs (MX formats + GPTQ/SmoothQuant) — https://arxiv.org/html/2405.07135v1 (accessed 2026-08-11)193- QuIP: 2-Bit Quantization of Large Language Models With Guarantees (Chee et al., NeurIPS 2023) — https://neurips.cc/virtual/2023/poster/69982 (accessed 2026-08-11)194- QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks (Tseng et al., ICML 2024) — slides — https://icml.cc/media/icml-2024/Slides/34816.pdf (accessed 2026-08-11)195- QuIP# codebase (Cornell RelaxML) — https://github.com/Cornell-RelaxML/quip-sharp (accessed 2026-08-11)196- QuIP# full text (PMC mirror, RVQ details) — https://pmc.ncbi.nlm.nih.gov/articles/PMC12395268 (accessed 2026-08-11)197- QTIP: Quantization with Trellises and Incoherence Processing (Tseng et al., NeurIPS 2024) — https://arxiv.org/html/2406.11235v1 (accessed 2026-08-11)198- Even Better, Even Faster Quantized LLMs with QTIP (Together AI blog) — https://www.together.ai/blog/even-better-even-faster-quantized-llms-with-qtip (accessed 2026-08-11)199- AQLM: Extreme Compression of Large Language Models via Additive Quantization (Egiazarian et al., ICML 2024) — https://arxiv.org/html/2401.06118v2 (accessed 2026-08-11)200- AQLM + PV-Tuning official repository — https://github.com/vahe1994/AQLM (accessed 2026-08-11)201- PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression (NeurIPS 2024) — https://proceedings.neurips.cc/paper_files/paper/2024/file/091166620a04a289c555f411d8899049-Paper-Conference.pdf (accessed 2026-08-11)202- The Evolution of Extreme LLM Compression: From QuIP to AQLM with PV-Tuning (Yandex, Medium) — https://medium.com/yandex/the-evolution-of-extreme-llm-compression-from-quip-to-aqlm-with-pv-tuning-19c44b91af96 (accessed 2026-08-11)203- GPTVQ: The Blessing of Dimensionality for LLM Quantization (van Baalen et al., Qualcomm) — https://arxiv.org/abs/2402.15319 (accessed 2026-08-11)204- GPTVQ repository — https://github.com/Qualcomm-AI-research/gptvq (accessed 2026-08-11)205- NestQuant: Nested Lattice Quantization for Matrix Products and LLMs (Savkin et al., ICML 2025) — https://arxiv.org/abs/2502.09720 (accessed 2026-08-11)206- Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compression (NeurIPS 2025) — https://neurips.cc/virtual/2025/poster/117396 (accessed 2026-08-11)207- SqueezeLLM: Dense-and-Sparse Quantization (Kim et al., ICML 2024) — https://arxiv.org/html/2306.07629v4 (accessed 2026-08-11)208- Half-Quadratic Quantization of Large Machine Learning Models (Mobius Labs, via Dropbox Tech) — https://dropbox.tech/machine-learning/halfquadratic-quantization-of-large-machine-learning-models (accessed 2026-08-11)209- QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (Ashkboos et al., NeurIPS 2024) — https://neurips.cc/virtual/2024/poster/94328 (accessed 2026-08-11)210- SpinQuant: LLM Quantization with Learned Rotations (Liu et al., ICLR 2025) — https://proceedings.iclr.cc/paper_files/paper/2025/file/e5b1c0d4866f72393c522c8a00eed4eb-Paper-Conference.pdf (accessed 2026-08-11)211- Rotation-based quantization with QuaRot (AMD Quark docs, R1–R4 rotations) — https://quark.docs.amd.com/release-0.9/pytorch/tutorial_quarot.html (accessed 2026-08-11)212- The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (Ma et al., Microsoft) — https://arxiv.org/abs/2402.17764 (accessed 2026-08-11)213- 1-bit AI Infra Part 1.1: Fast and Lossless BitNet b1.58 Inference on CPUs (bitnet.cpp) — https://arxiv.org/html/2410.16144v1 (accessed 2026-08-11)214- Bitnet.cpp: Efficient Edge Inference for Ternary LLMs (ACL 2025; TL/I2_S kernels, M2 Ultra 100B result) — https://aclanthology.org/2025.acl-long.457.pdf (accessed 2026-08-11)215- microsoft/BitNet official inference framework — https://github.com/microsoft/BitNet (accessed 2026-08-11)216- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (Huang et al., ICML 2024) — https://github.com/Aaronhuang-778/BiLLM (accessed 2026-08-11)217- ParetoQ: Scaling Laws in Extremely Low-bit LLM Quantization (Liu et al., Meta, NeurIPS 2025) — https://arxiv.org/html/2502.02631v2 (accessed 2026-08-11)218- ParetoQ (PyTorch blog) — https://pytorch.org/blog/paretoq-scaling-laws-in-extremely-low-bit-llm-quantization (accessed 2026-08-11)219- EfficientQAT: Efficient Quantization-Aware Training for LLMs (Chen et al., ACL 2025) — https://arxiv.org/abs/2407.11062 (accessed 2026-08-11)220- EfficientQAT repository (w2g64 PPL tables) — https://github.com/OpenGVLab/EfficientQAT (accessed 2026-08-11)221- Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs (Park et al., ICML 2024 oral) — https://arxiv.org/html/2402.10517v4 (accessed 2026-08-11)222- Any-Precision LLM repository (bitplane engine) — https://github.com/SNU-ARC/any-precision-llm (accessed 2026-08-11)223- Matryoshka Quantization (Nair et al., Google DeepMind, ICLR 2025 oral) — https://iclr.cc/virtual/2025/10000114 (accessed 2026-08-11)224- Matryoshka Quantization topic overview (Emergent Mind) — https://www.emergentmind.com/topics/matryoshka-quantization-matquant (accessed 2026-08-11)225- Multi-Bitwidth Quantization for LLMs Using Additive Codebooks ("Drop-by-Drop", successive refinement) — https://arxiv.org/html/2606.12876v1 (accessed 2026-08-11)226- Progressive Mixed-Precision Decoding for Efficient LLM Inference (Chen et al., ICLR 2025) — https://arxiv.org/abs/2410.13461 (accessed 2026-08-11)227- Mixed-Precision Quantization for Language Models (survey, Oct 2025; PMDP/MPMLC taxonomy) — https://arxiv.org/html/2510.16805v1 (accessed 2026-08-11)228- SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators (Apple ML Research) — https://machinelearning.apple.com/research/seedlm-compressing (accessed 2026-08-11)229- SeedLM (arXiv full text) — https://arxiv.org/html/2410.10714v1 (accessed 2026-08-11)230- DFloat11: 70% Size, 100% Accuracy — Lossless LLM Compression via Dynamic-Length Float — https://huggingface.co/papers/2504.11651 (accessed 2026-08-11)231- DFloat11 repository (LeanModels, NeurIPS 2025) — https://github.com/LeanModels/DFloat11 (accessed 2026-08-11)232- DFloat11 throughput caveats (Hacker News discussion incl. appendix numbers) — https://news.ycombinator.com/item?id=43796935 (accessed 2026-08-11)233- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (Liu et al., ICML 2024) — https://arxiv.org/abs/2402.02750 (accessed 2026-08-11)234- KIVI repository — https://github.com/jy-yuan/KIVI (accessed 2026-08-11)235- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (Hooper et al., NeurIPS 2024) — https://arxiv.org/abs/2401.18079 (accessed 2026-08-11)236- KV Cache is 1 Bit Per Channel: Coupled Quantization (NeurIPS 2024) — https://proceedings.neurips.cc/paper_files/paper/2024/file/05d6b5b6901fb57d2c287e1d3ce6d63c-Paper-Conference.pdf (accessed 2026-08-11)237- mlx.core.quantize documentation (affine/mxfp4/mxfp8/nvfp4 modes) — https://ml-explore.github.io/mlx/build/html/python/_autosummary/mlx.core.quantize.html (accessed 2026-08-11)238- mlx.nn.quantize documentation (quantize_input, class predicates) — https://ml-explore.github.io/mlx/build/html/python/_autosummary/mlx.nn.quantize.html (accessed 2026-08-11)239- mlx-lm LEARNED_QUANTS.md (DWQ, dynamic_quant, AWQ/GPTQ recipes) — https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/LEARNED_QUANTS.md (accessed 2026-08-11)240- MLX Quantization on Apple Silicon: dynamic_quant vs AWQ vs GPTQ vs DWQ (Hannecke, Medium) — https://medium.com/@michael.hannecke/mlx-quantization-on-apple-silicon-dynamic-quant-vs-awq-vs-gptq-vs-dwq-8b2a5af2b53f (accessed 2026-08-11)241- Better inference quality and performance for MLX on Apple Silicon (Feldman; K-quant vs MLX affine KL measurements) — https://www.linkedin.com/pulse/better-inference-quality-performance-mlx-apple-silicon-asher-feldman-ztm0e (accessed 2026-08-11)242- Very slow IQ quant performance on Apple Silicon (llama.cpp discussion #5617, ikawrakow measurements) — https://github.com/ggml-org/llama.cpp/discussions/5617 (accessed 2026-08-11)243- Overview of GGUF quantization methods (r/LocalLLaMA; i-quant LUT bottleneck notes) — https://www.reddit.com/r/LocalLLaMA/comments/1ba55rj/overview_of_gguf_quantization_methods (accessed 2026-08-11)244- LLM Quantization Formats Compared: GGUF vs MLX vs EXL3 vs GPTQ vs AWQ vs FP8 (D-Central; format inventories) — https://d-central.tech/llm-quantization-formats (accessed 2026-08-11)245- GGUF vs MLX Quantization Formats on Apple Silicon (Contra Collective, 2026) — https://contracollective.com/blog/gguf-vs-mlx-quantization-formats-apple-silicon-2026 (accessed 2026-08-11)246- llama.cpp Metal Backend vs MLX: Compute Path Comparison (Contra Collective, 2026) — https://contracollective.com/blog/llama-cpp-metal-vs-mlx-backend-apple-silicon-2026 (accessed 2026-08-11)247- llama.cpp supports gpt-oss in native MXFP4 (discussion #15095) — https://github.com/ggml-org/llama.cpp/discussions/15095 (accessed 2026-08-11)248- exllamav3 / EXL3 trellis format (turboderp) — https://github.com/turboderp-org/exllamav3 (accessed 2026-08-11)249- KV Cache and Context Length on Apple Silicon (Contra Collective, 2026; llama.cpp/mlx-lm KV flags) — https://contracollective.com/blog/kv-cache-context-length-apple-silicon-local-inference-2026 (accessed 2026-08-11)250- Running LLMs locally on a Mac (MacKinlay; KV cache quant flags across runtimes) — https://danmackinlay.name/notebook/local_llm_mac.html (accessed 2026-08-11)251- KVSplit: differentiated K/V precision on Apple Silicon (Show HN) — https://news.ycombinator.com/item?id=44009321 (accessed 2026-08-11)252- TurboQuant — Extreme KV Cache Quantization with Metal kernels (llama.cpp discussion #20969) — https://github.com/ggml-org/llama.cpp/discussions/20969 (accessed 2026-08-11)253- M2Cache: Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Caching — https://arxiv.org/abs/2410.14740 (accessed 2026-08-11)254- SSD Offloading for LLM MoE Weights Considered Harmful in Energy Efficiency — https://www.alphaxiv.org/overview/2508.06978 (accessed 2026-08-11)255- Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices (MLX Q4-KV state of play) — https://arxiv.org/html/2603.04428v1 (accessed 2026-08-11)256- MLX vs llama.cpp on Apple Silicon: Benchmarks, M5 Neural Accelerators, Ollama switch — https://yage.ai/share/mlx-apple-silicon-en-20260331.html (accessed 2026-08-11)257