SPB Git

spb/localvm-research Public License

Running LLMs larger than memory on a consumer Mac — falsification-driven research: margin-gated deferred refinement, out-of-core verification on Apple Silicon. TR-01 published.

Python 63.2% JavaScript 23.5% CSS 11.8% Shell 0.9% Makefile 0.5%

README: comprehensive English rewrite — logo, badges, headline metrics, campaign table, reproduction guide, citation

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simon-Pierre Boucher committed 1 h ago (Aug 12, 2026) parent d5fb420

Showing 1 changed file with +215 and −54

modified README.md +215 −54
@@ -5,66 +5,227 @@ author: Simon-Pierre Boucher
5 5 contact: contact@spboucher.ai
6 6 created: 2026-08-11
7 7 modified: 2026-08-12
8 −status: draft
8 +status: reviewed
9 9 ---
10 10
11 +<div align="center">
12 +
13 +<img src="web/public/logo.svg" width="96" alt="localvm-research logo">
14 +
11 15 # localvm-research
12 16
13 −**Can an existing pretrained LLM that does not fit in a consumer Mac's memory be
14 −transformed post-training into an execution representation whose instantaneous
15 −working set is dramatically smaller than the full checkpoint — while preserving
16 −most of its useful capabilities?**
17 +**Running LLMs larger than memory on a consumer Mac**
18 +
19 +[![status](https://img.shields.io/badge/status-active_research-2a78d6)](https://www.localvm.dev)
20 +[![platform](https://img.shields.io/badge/platform-Apple_Silicon_·_macOS-0b0b0b)](https://www.localvm.dev/about)
21 +[![stack](https://img.shields.io/badge/stack-MLX_0.32_·_Metal_·_Python_3.14-1baf7a)](https://github.com/ml-explore/mlx)
22 +[![experiments](https://img.shields.io/badge/experiments-8_completed_·_3_refuted-eb6834)](https://www.localvm.dev/experiments)
23 +[![sources](https://img.shields.io/badge/literature-304_sources-4a3aa7)](https://www.localvm.dev/doc/research/bibliography.md)
24 +[![report](https://img.shields.io/badge/publication-TR--01-2a78d6)](https://www.localvm.dev/publications)
25 +[![license](https://img.shields.io/badge/license-all_rights_reserved-52514e)](LICENSE)
26 +
27 +**Live research platform: [www.localvm.dev](https://www.localvm.dev)** — every hypothesis, result, figure, and failure, browsable.
17 28
18 −This repository is a living research project (charter: [CLAUDE.md](CLAUDE.md)).
19 −The central distinction under investigation:
29 +*Author: **Simon-Pierre Boucher** · [contact@spboucher.ai](mailto:contact@spboucher.ai)*
30 +
31 +</div>
32 +
33 +---
34 +
35 +## The question
36 +
37 +> Can an existing pretrained LLM whose memory requirements exceed a consumer
38 +> Apple Silicon Mac be transformed **post-training** into an execution
39 +> representation whose *instantaneous working set* is dramatically smaller
40 +> than the full checkpoint — while preserving most of its capabilities?
41 +
42 +The project's central distinction, which everything here tries to decouple:
20 43
21 44 ```text
22 −total model size ≠ resident model size ≠ bytes read per token ≠ parameters required for this token
45 +total model size ≠ resident model size ≠ bytes read per token ≠ parameters required for THIS token
23 46 ```
24 47
25 −- **Author:** Simon-Pierre Boucher — <contact@spboucher.ai>
26 −- **Primary platform:** Apple Silicon Mac (unified memory, Metal, internal NVMe), macOS 14+
27 −- **Status:** Phases 1–4 complete (≈300-source literature sweep → state-of-the-art map →
28 − 24 falsifiable gaps → ranked candidates). Micro-experiment campaign complete
29 − (8 experiments: 3 confirmed, 3 usefully refuted, plus substrate + prototype).
30 − A candidate architecture — **margin-gated deferred refinement with a layer-streamed
31 − verifier** — was selected by the evidence and prototyped. See `research/LOG.md` for the
32 − full audit trail and https://www.localvm.dev for the browsable paper trail.
33 −
34 −## Headline measured results (Apple M5 Max, 48 GB, macOS 27)
35 −
36 −- **SSD substrate (expH):** internal NVMe ceiling ≈ 13.1 GB/s (iostat-validated);
37 − 67 MB/s at 4 KiB QD1 → 13.8 GB/s at 1 MiB QD8; Metal GPU load costs < 5%.
38 − `F_NOCACHE` does not bypass resident pages — cold benchmarks must write uncached.
39 −- **Decision stability (expG):** low-bit top-1 margin predicts disagreement with the
40 − full-precision model (AUROC 0.85–0.98); improves with scale (esc@99%: 36.6% at 1.7B →
41 − 22.6% at 8B, 4-bit). Naive affine 2-bit destroys the model (3% agreement).
42 −- **Negative results (expF/expA/expB):** quantization damage is diffuse across depth;
43 − SwiGLU FFN importance has no pageable block structure and no temporal working set —
44 − dynamic sparsity paging (DejaVu / LLM-in-a-flash paradigm) measurably does not
45 − transfer to dense modern architectures.
46 −- **Prototype (candidate_01, 1.7B):** q4 resident + windowed q8 verification sweeps →
47 − output quality statistically equal to resident-q8 (bf16 judge) at 237 MB/token
48 − streamed (9× under the checkpoint), with a runtime knob (τ) trading bytes for quality.
49 −- **Scale run (32B on 48 GB):** q8-32B (34.8 GB) cannot co-reside with the q4 base —
50 − it is layer-streamed per sweep at ~11.6 GB/s sequential; verified output beats the
51 − only fitting alternative (pure q4) by +0.28 nats (8B judge) at 3.7 GB/token
52 − (9.4× under the checkpoint), 1.69 tok/s (prototype speed; headroom documented).
53 −
54 −## Layout
55 −
56 −| Path | Contents |
57 −|---|---|
58 −| `research/` | Scientific paper trail: log, state of the art, gaps, rankings, bibliography |
59 −| `src/localvm/` | Core library (models, transforms, runtime, backends, instrumentation, quality) |
60 −| `experiments/` | Micro-experiments (A–H) and candidate prototypes |
61 −| `benchmarks/` | Unified harness, hardware manifest, baselines, eval datasets |
62 −| `results/` | Raw + aggregated results (reproducible from commit + config + seed + manifest) |
63 −| `tools/` | `check_headers.py`, `new_experiment.py`, `report.py` |
64 −
65 −## Reproducibility
66 −
67 −Every result is reproducible from: commit hash + config + seed + hardware manifest
68 −(`benchmarks/hardware_manifest.py`). No result is reported from an uncommitted tree.
69 −
70 −*This README is updated continuously and finalized last (charter §18).*
48 +This is a falsification-driven research project run under a strict charter
49 +([CLAUDE.md](CLAUDE.md)): hypotheses are registered with explicit kill criteria
50 +*before* experiments run, no result is reported from an uncommitted tree, every
51 +result file embeds a hardware manifest, and **negative results are kept and
52 +published**. The full audit trail lives in the append-only
53 +[research log](research/LOG.md).
54 +
55 +## Headline results
56 +
57 +All measured on the primary research machine — **Apple M5 Max, 6P+12E CPU,
58 +40-core GPU, 48 GB unified memory, 2 TB AP2048Z NVMe, macOS 27.0**.
59 +
60 +| # | Result | Key numbers |
61 +|---|--------|-------------|
62 +| 1 | **SSD substrate envelope** (expH) | device ceiling ≈ **13.1 GB/s** (iostat-validated); 4 KiB QD1 random = **67 MB/s** → 1 MiB QD8 = **13.8 GB/s** (200× spread); saturated Metal GPU costs **< 5%** of SSD throughput |
63 +| 2 | **Margins predict disagreement** (expG) | AUROC **0.85–0.98** (3/4/8-bit); domain-uniform; escalation for 99% agreement falls with scale: **36.6% → 22.6%** (1.7B → 8B, 4-bit) |
64 +| 3 | **Naive 2-bit is dead** (expG) | affine 2-bit: **3% greedy agreement**, KL ≈ 11.9 — the 2–3-bit cliff, measured |
65 +| 4 | **Progressive residual coding works** (expD) | hidden-state error **÷5 per stage**; two-tier margin policy: **97.6% agreement @ 35% escalation** |
66 +| 5 | **Three escalation routes refuted** (expF/A/B) | layer sensitivity spread only **1.5×**; 95% FFN energy needs **77%** of 64-neuron blocks; 128-token working-set union = **99.9%** — dynamic sparsity paging does *not* transfer to dense SwiGLU models |
67 +| 6 | **Quality–bytes continuum** (candidate, 1.7B) | verify-all output **statistically = resident-q8** (bf16 judge −0.1297 vs −0.1328) at **237 MB/token** streamed — **9× under the checkpoint**; τ knob: 150/211/237 MB/token ↔ 47%/83%/~100% of the q4→q8 gap |
68 +| 7 | **Out-of-core at 32B** (candidate, scale) | q8-32B (34.8 GB) **cannot co-reside** with the q4 base on 48 GB — streamed layer-by-layer at **~11.6 GB/s**; **+0.28 nats** over pure-q4 (the only fitting config) at **3.7 GB/token** (9.4× under checkpoint), 1.69 tok/s |
69 +| 8 | **Two macOS measurement traps** | `F_NOCACHE` does **not** evict resident pages (naïve "cold" benchmarks read RAM at 42–115 GB/s); Metal prefill vs incremental decode flips **1.56%/token** of a model's own greedy argmax — token-exact reproduction is not a coherent target |
70 +
71 +## The architecture — margin-gated deferred refinement
72 +
73 +Selected by the evidence (not by preference — see the
74 +[candidate ranking](research/candidate_ranking.md) and three refutations above):
75 +
76 +1. **Resident base** — the q4 quantization of the model lives in unified
77 + memory and generates greedily, recording each decision's **top-1 logit
78 + margin** (a free byproduct of the forward pass).
79 +2. **Deferral** — decisions with margin < τ are flagged; generation never blocks.
80 +3. **Amortized verification sweeps** — every W tokens, the *same checkpoint's*
81 + q8 variant teacher-forces the window in one pass through a persistent
82 + incremental verify-KV-cache. When q8 cannot co-reside (the target regime),
83 + it is **streamed layer-by-layer from SSD** (materialize → compute →
84 + re-lazify) — one *sequential* pass over the checkpoint per sweep, exactly
85 + the access pattern the measured SSD envelope rewards.
86 +4. **Rollback** — on the first checked flip the runtime trims both KV caches,
87 + accepts the q8 token, and resumes.
88 +
89 +**τ is a runtime knob that prices output quality in verification bytes** — a
90 +property no static quantization format offers.
91 +
92 +Implementation: [`experiments/candidate_01/implementation/`](experiments/candidate_01/implementation/)
93 +(`runtime.py` — generation/deferral/rollback loop; `streaming_verifier.py` —
94 +per-layer weight streaming on unified memory).
95 +
96 +## The experimental campaign
97 +
98 +Every experiment has a preregistered `hypothesis.md` (seven-field scientific
99 +block with falsification criterion) and a post-run `analysis.md`.
100 +
101 +| Experiment | Question | Verdict |
102 +|------------|----------|---------|
103 +| [expH — SSD feasibility](experiments/micro/expH_ssd_feasibility/) | Can the internal NVMe sustain weight-streaming rates? | ✅ **Confirmed** — with a strict ≥256 KiB / QD≥4 / sequential contract |
104 +| [expG — decision stability](experiments/micro/expG_decision_stability/) | Does the cheap pass's margin predict disagreement? | ✅ **Confirmed** (AUROC 0.85–0.98, improves with scale) |
105 +| [expD — progressive reconstruction](experiments/micro/expD_progressive_reconstruction/) | Do residual ladders converge? Does gating compose? | ✅ **Confirmed** (with an honest static-parity caveat) |
106 +| [expF — error accumulation](experiments/micro/expF_error_accumulation/) | Is quantization damage concentrated in layers? | ❌ **Refuted** — diffuse (1.5× spread), repair is cooperative |
107 +| [expA — weight concentration](experiments/micro/expA_weight_concentration/) | Is per-token FFN importance pageable at block granularity? | ❌ **Refuted** — neuron-level concentration exists but is scattered |
108 +| [expB — token stability](experiments/micro/expB_token_stability/) | Is the important-block set temporally cacheable? | ❌ **Refuted** — union ≈ 100% within 128 tokens; Jaccard ≈ random null |
109 +| [candidate_01 — 1.7B](experiments/candidate_01/) | Does the assembled runtime deliver a quality–bytes continuum? | ✅ q8-equivalent quality at 9× fewer bytes |
110 +| [candidate_01 — 32B scale](experiments/candidate_01/benchmark_scale.py) | Does it work when the verifier physically cannot fit? | ✅ +0.28 nats over the only fitting alternative |
111 +
112 +The refutations are load-bearing: they killed the fashionable design (dynamic
113 +sparsity paging à la DejaVu / LLM-in-a-flash) *on modern dense architectures*
114 +with numbers, and left batch-amortized verification as the only surviving
115 +route — which the prototype then validated.
116 +
117 +## Novelty, honestly stated
118 +
119 +An adversarial prior-art search ([novelty_check.md](research/novelty_check.md),
120 +45 sources) found **every component anticipated** — precision-tiered
121 +draft/verify (QSpec, Apple QuantSpec), resident-draft + flash-streamed target
122 +(Lever 2026, CATS 2026), uncertainty-gated verification (U-HLM/UGSD, vs cloud),
123 +window-amortized verification (SpecExec), MLX layer streaming (mlx-flash).
124 +Four narrow claims survived: **(N1)** margin-gating of the *same model's*
125 +higher-precision **weight-streaming I/O** on one machine; **(N2)**
126 +bytes-streamed-per-token vs checkpoint size as the explicit objective, with τ
127 +as the knob; **(N3)** the training-free same-checkpoint q4/q8 split
128 +demonstrated where the verifier cannot reside; **(N4)** the
129 +unified-memory/Metal instantiation. Attribution obligations are recorded and
130 +the check should be re-run before any external submission.
131 +
132 +## Repository structure
133 +
134 +```text
135 +localvm-research/
136 +├── CLAUDE.md # research charter (methodology contract)
137 +├── research/ # the scientific paper trail
138 +│ ├── LOG.md # append-only research log (every decision)
139 +│ ├── state_of_the_art.md # Phase 2 — technique taxonomy, overlap analysis
140 +│ ├── research_gaps.md # Phase 3 — 24 falsifiable gaps (G01–G24)
141 +│ ├── candidate_ranking.md # Phase 4 — 10-axis scoring, 4 candidates
142 +│ ├── novelty_check.md # Phase 11 — adversarial prior-art verdict
143 +│ ├── bibliography.md # 304 sources with URLs and access dates
144 +│ └── notes/ # 5 Phase-1 literature theme notes
145 +├── experiments/
146 +│ ├── micro/expA…expH/ # preregistered micro-experiments
147 +│ └── candidate_01/ # the prototype runtime + 1.7B/32B benchmarks
148 +├── benchmarks/
149 +│ ├── hardware_manifest.py # macOS-native fingerprint (embedded in results)
150 +│ └── datasets/eval_prompts.json # 6-domain eval prompts (code/math/chat/FR/EN/reasoning)
151 +├── results/ # raw JSON results (commit + config + seed + manifest)
152 +├── src/localvm/ # library (quality/decision_stats.py, backends, …)
153 +├── tools/ # check_headers.py, new_experiment.py, make_pub_figures.py
154 +├── docs/publications/ # official technical reports (TR-01) + SVG figures
155 +└── web/ # the public research platform (Express, deployed)
156 +```
157 +
158 +## Reproduce it
159 +
160 +```bash
161 +# 1. Environment (Apple Silicon Mac, macOS 14+)
162 +python3 -m venv .venv
163 +.venv/bin/pip install mlx mlx-lm numpy
164 +
165 +# 2. Fingerprint your machine (embedded in every result)
166 +python3 benchmarks/hardware_manifest.py
167 +
168 +# 3. Substrate envelope (~15 min, writes results/expH_ssd_feasibility/)
169 +cd experiments/micro/expH_ssd_feasibility && ../../../.venv/bin/python benchmark.py
170 +
171 +# 4. Decision stability (~10 min, downloads Qwen3-1.7B-bf16)
172 +cd ../expG_decision_stability && ../../../.venv/bin/python benchmark.py
173 +
174 +# 5. The prototype (build quantized artifacts once, then run)
175 +cd ../../candidate_01
176 +../../.venv/bin/python benchmark.py --build
177 +../../.venv/bin/python benchmark.py # 1.7B continuum
178 +../../.venv/bin/python benchmark_scale.py # 32B out-of-core (downloads ~52 GB)
179 +
180 +# 6. Regenerate all publication figures from committed results
181 +python3 tools/make_pub_figures.py
182 +
183 +# Quality gates
184 +python3 tools/check_headers.py # mandatory header check (CI-style)
185 +```
186 +
187 +Every benchmark writes `results/<experiment>/<UTC timestamp>/results.json`
188 +with the full config and hardware manifest embedded.
189 +
190 +## Publications
191 +
192 +- **[TR-01 — Margin-Gated Deferred Refinement: Streaming Higher-Precision LLM
193 + Quality Than Fits in Memory on Consumer Apple Silicon](docs/publications/TR-01-margin-gated-deferred-refinement.md)**
194 + (August 2026) — the complete write-up with figures, prior-art attribution,
195 + and limitations. Also on the platform: [localvm.dev/publications](https://www.localvm.dev/publications).
196 +
197 +## Known limitations & roadmap
198 +
199 +Recorded in TR-01 §7 and the research log — the honest list: prototype speed
200 +(1.69 tok/s at 32B) is below interactive; no resident-q5/q6 baseline at 32B
201 +yet (they fit in 48 GB and may dominate at that scale — the architecture's
202 +necessary regime is 70B-class, untested); the 8B judge is a proxy with no
203 +error bars at n=12; no task-level benchmarks yet; one model family, one
204 +machine. **Next:** resident q5/q6 baselines → task benchmarks (GSM8K-class) →
205 +τ calibration on real quantized-artifact margins → 70B point → TR-02.
206 +
207 +## Author & citation
208 +
209 +**Simon-Pierre Boucher** — [contact@spboucher.ai](mailto:contact@spboucher.ai)
210 +
211 +```bibtex
212 +@techreport{boucher2026localvm,
213 + author = {Boucher, Simon-Pierre},
214 + title = {Margin-Gated Deferred Refinement: Streaming Higher-Precision
215 + LLM Quality Than Fits in Memory on Consumer Apple Silicon},
216 + institution = {localvm-research},
217 + number = {TR-01},
218 + year = {2026},
219 + month = {August},
220 + url = {https://www.localvm.dev/publications}
221 +}
222 +```
223 +
224 +Citation metadata: [CITATION.cff](CITATION.cff).
225 +
226 +## License
227 +
228 +All rights reserved (research code) — © 2026 Simon-Pierre Boucher.
229 +See [LICENSE](LICENSE). The research trail is public for reading and
230 +verification at [www.localvm.dev](https://www.localvm.dev); reuse requires
231 +written authorization.
71 232