SPB Git

spb/localvm-research Public License

Running LLMs larger than memory on a consumer Mac — falsification-driven research: margin-gated deferred refinement, out-of-core verification on Apple Silicon. TR-01 published.

Python 63.2% JavaScript 23.5% CSS 11.8% Shell 0.9% Makefile 0.5%

candidate_01 32B scale run: charter 16.A/B demonstrated in prototype form

q8-32B (does not fit beside base on 48 GB) streamed at ~11.6 GB/s
sequential; +0.28 nats over pure-q4 (only fitting alternative);
3.72 GB/token logical (9.4x under checkpoint); 1.69 tok/s with
measured headroom.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simon-Pierre Boucher committed 4 h ago (Aug 12, 2026) parent 30886de

Showing 5 changed files with +324 and −0

modified experiments/candidate_01/analysis.md +43 −0
@@ -108,3 +108,46 @@ Findings
108 108
109 109 Next: scale run at 32B (q8 does not fit comfortably wired on this Mac) —
110 110 bytes/token, tok/s, judge quality vs the only real alternative (pure q4).
111 +
112 +## Addendum — scale run at 32B: the target regime (2026-08-12)
113 +
114 +Run: `results/candidate_01_scale32b/20260812T055506Z/` · Qwen3-32B on the
115 +48 GB M5 Max: q4 (17.5 GB) resident; q8 (34.8 GB) **cannot be co-resident** —
116 +each sweep streams it layer-by-layer from SSD (materialize → compute →
117 +re-lazify), reading the checkpoint sequentially at ~11.6 GB/s (≈ the expH
118 +device ceiling). 12 prompts × 96 tokens, W=32. Judge: independent Qwen3-8B
119 +bf16 (the 32B bf16 obviously cannot run here).
120 +
121 +```text
122 +Config judge logprob tok/s sweep GB/token (logical)
123 +pure q4 (resident) -0.7506 ~25 — 0
124 +margin τ=2.0 -0.5460 1.62 3.0 s 3.93
125 +verify-all -0.4667 1.69 3.0 s 3.72
126 +(q8 resident: impossible on this machine — that is the point)
127 +```
128 +
129 +Findings
130 +1. CHARTER §16.A/§16.B DEMONSTRATED IN PROTOTYPE FORM: a quality level
131 + whose weights do not fit in unified memory (q8-32B; q4+q8 = 52 GB
132 + total representation on a 48 GB machine) is made usable locally.
133 + Verified output improves the 8B-judge score by 0.28 nats over the
134 + only configuration that actually fits (pure q4) — 38% relative — and
135 + bytes streamed per token (3.7 GB) are 9.4× below the streamed
136 + checkpoint, with a strictly sequential SSD pattern at device ceiling.
137 +2. Throughput 1.65 tok/s end-to-end — batch-usable, below interactive.
138 + The cost structure is now measured: 3.0 s/sweep (SSD-bound, ~1 sweep
139 + per ≤32 tokens plus rollback-triggered re-sweeps) + 25 tok/s base
140 + decode + rollback regeneration. Identified headroom, in order:
141 + (a) larger windows (W=64/128 halves/quarters GB/token and sweep
142 + frequency); (b) accepting q8 token runs directly from sweep logits
143 + (free speculative continuation — rollbacks currently regenerate at
144 + base speed then re-sweep); (c) overlapping sweep I/O with continued
145 + base generation (expH: GPU compute costs SSD reads <5%).
146 +3. Honest limits: quality has no measurable ceiling on this machine
147 + (no resident q8 to compare against); the 8B judge is an imperfect
148 + proxy; W=32 was tuned for nothing — no sweep/window search was done;
149 + 1.65 tok/s is prototype speed, not a system claim.
150 +
151 +Next: Phase 11 novelty check on the assembled architecture (margin-gated
152 +deferral + layer-streamed amortized verification on unified memory), then
153 +either the throughput levers above or the 14B/70B scale points.
added experiments/candidate_01/scale32b_full.log +52 −0
@@ -0,0 +1,52 @@
1 + Fetching 14 files: 0%| | 0/14 [00:00<?, ?it/s] Fetching 14 files: 100%|██████████| 14/14 [00:00<00:00, 6875.91it/s]
2 + Fetching 17 files: 0%| | 0/17 [00:00<?, ?it/s] Fetching 17 files: 100%|██████████| 17/17 [00:00<00:00, 7145.32it/s]
3 +loading q4 resident …
4 +q8 checkpoint (streamed): 34.8 GB
5 +baseline: pure q4 …
6 + 1/12 (96 tok, 24.1 tok/s)
7 + 2/12 (96 tok, 26.7 tok/s)
8 + 3/12 (96 tok, 26.5 tok/s)
9 + 4/12 (96 tok, 26.6 tok/s)
10 + 5/12 (96 tok, 26.6 tok/s)
11 + 6/12 (96 tok, 26.6 tok/s)
12 + 7/12 (96 tok, 26.7 tok/s)
13 + 8/12 (96 tok, 26.8 tok/s)
14 + 9/12 (96 tok, 26.7 tok/s)
15 + 10/12 (96 tok, 26.5 tok/s)
16 + 11/12 (96 tok, 26.5 tok/s)
17 + 12/12 (96 tok, 26.5 tok/s)
18 +runtime: mode=margin tau=2.0 W=32 …
19 + 1/12 (96 tok, 5 sweeps, 3 rollbacks, last sweep io 2.9s)
20 + 2/12 (96 tok, 11 sweeps, 9 rollbacks, last sweep io 2.8s)
21 + 3/12 (96 tok, 4 sweeps, 1 rollbacks, last sweep io 2.9s)
22 + 4/12 (96 tok, 7 sweeps, 5 rollbacks, last sweep io 2.8s)
23 + 5/12 (96 tok, 15 sweeps, 15 rollbacks, last sweep io 3.0s)
24 + 6/12 (96 tok, 16 sweeps, 15 rollbacks, last sweep io 2.8s)
25 + 7/12 (96 tok, 16 sweeps, 15 rollbacks, last sweep io 2.9s)
26 + 8/12 (96 tok, 20 sweeps, 19 rollbacks, last sweep io 2.8s)
27 + 9/12 (96 tok, 4 sweeps, 2 rollbacks, last sweep io 3.0s)
28 + 10/12 (96 tok, 11 sweeps, 10 rollbacks, last sweep io 3.0s)
29 + 11/12 (96 tok, 10 sweeps, 8 rollbacks, last sweep io 2.9s)
30 + 12/12 (96 tok, 11 sweeps, 10 rollbacks, last sweep io 2.9s)
31 + tok/s=1.62 sweepLat=3.0s GB/token(logical)=3.93
32 +runtime: mode=verify-all tau=2.0 W=32 …
33 + 1/12 (96 tok, 5 sweeps, 3 rollbacks, last sweep io 3.0s)
34 + 2/12 (96 tok, 11 sweeps, 9 rollbacks, last sweep io 2.8s)
35 + 3/12 (96 tok, 4 sweeps, 1 rollbacks, last sweep io 2.8s)
36 + 4/12 (96 tok, 5 sweeps, 3 rollbacks, last sweep io 3.0s)
37 + 5/12 (96 tok, 15 sweeps, 15 rollbacks, last sweep io 3.0s)
38 + 6/12 (96 tok, 15 sweeps, 15 rollbacks, last sweep io 2.8s)
39 + 7/12 (96 tok, 16 sweeps, 15 rollbacks, last sweep io 2.9s)
40 + 8/12 (96 tok, 14 sweeps, 13 rollbacks, last sweep io 2.9s)
41 + 9/12 (96 tok, 7 sweeps, 5 rollbacks, last sweep io 3.0s)
42 + 10/12 (96 tok, 11 sweeps, 10 rollbacks, last sweep io 3.0s)
43 + 11/12 (96 tok, 10 sweeps, 8 rollbacks, last sweep io 2.9s)
44 + 12/12 (96 tok, 10 sweeps, 9 rollbacks, last sweep io 2.8s)
45 + tok/s=1.69 sweepLat=3.0s GB/token(logical)=3.72
46 +freeing 32B models; loading 8B bf16 judge …
47 + Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s] Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 3096.00it/s]
48 + pure_q4: -0.7506
49 + margin_tau2.0: -0.5460
50 + verify-all_tau2.0: -0.4667
51 +
52 +wrote /Users/simon-pierreboucher/Desktop/localvm-research/results/candidate_01_scale32b/20260812T055506Z/results.json
modified research/LOG.md +22 −0
@@ -287,3 +287,25 @@ Format per entry: date/time (local, with timezone) · question · experiment ·
287 287 from sweep logits — free speculative continuation).
288 288 - **Decision:** Scale run at 32B (q4 17 GB resident, q8 35 GB streamed — q8 does not fit
289 289 wired on 48 GB): the regime the architecture exists for. Artifacts downloading.
290 +
291 +---
292 +
293 +## 2026-08-12 16:40 EDT — 32B scale run: charter §16.A/B demonstrated in prototype form
294 +
295 +- **Question:** Does the architecture deliver in its target regime — a model configuration
296 + that does not fit in unified memory?
297 +- **Experiment:** Qwen3-32B on the 48 GB M5 Max: q4 resident (17.5 GB) + q8 (34.8 GB)
298 + layer-streamed per sweep via StreamingVerifier (materialize→compute→re-lazify; sequential
299 + SSD reads at ~11.6 GB/s ≈ expH ceiling). 12 prompts × 96 tokens.
300 + Results: `results/candidate_01_scale32b/20260812T055506Z/` (smoke: 20260812T053042Z).
301 +- **Result:** verify-all: judge (8B bf16) -0.4667 vs pure-q4 -0.7506 (+0.28 nats, 38% rel.);
302 + 1.69 tok/s; 3.72 GB/token logical (9.4× under streamed checkpoint); sweeps 3.0 s. Margin
303 + τ=2.0 intermediate (-0.546, 1.62 tok/s). q8 cannot run resident on this machine at all.
304 +- **Interpretation:** Total representation (52 GB) exceeds unified memory (48 GB) yet its
305 + quality is usable locally — bytes-per-token decoupled from checkpoint size with a strictly
306 + sequential access pattern. Below interactive (1.65 tok/s): measured headroom = larger
307 + windows, speculative acceptance of q8 runs from sweep logits, I/O-compute overlap (<5%
308 + contention per expH). Limits recorded: no q8 ceiling measurable here; 8B judge is a proxy;
309 + no window tuning.
310 +- **Decision:** Phase 11 novelty check on the assembled architecture next; then throughput
311 + levers or additional scale points.
added results/candidate_01_scale32b/20260812T053042Z/results.json +92 −0
@@ -0,0 +1,92 @@
1 +{
2 + "experiment": "candidate_01_scale32b",
3 + "author": "Simon-Pierre Boucher",
4 + "contact": "contact@spboucher.ai",
5 + "manifest": {
6 + "author": "Simon-Pierre Boucher",
7 + "contact": "contact@spboucher.ai",
8 + "project": "localvm-research",
9 + "collected_utc": "2026-08-12T05:30:42.799916+00:00",
10 + "chip": {
11 + "brand": "Apple M5 Max",
12 + "arch": "arm64",
13 + "cores_total": 18,
14 + "cores_performance": 6,
15 + "cores_efficiency": 12,
16 + "gpu_cores": 40
17 + },
18 + "memory": {
19 + "unified_bytes": 51539607552,
20 + "unified_gb": 48.0,
21 + "pagesize": 16384
22 + },
23 + "ssd": {
24 + "model": "APPLE SSD AP2048Z",
25 + "size": "2 TB",
26 + "smart_status": "Verified"
27 + },
28 + "os": {
29 + "product": "macOS",
30 + "version": "27.0",
31 + "build": "26A5388g",
32 + "kernel": "27.0.0"
33 + },
34 + "software": {
35 + "python": "3.14.4",
36 + "mlx": "0.32.0",
37 + "mlx_lm": "0.31.3",
38 + "torch": null,
39 + "numpy": "2.5.2"
40 + },
41 + "git": {
42 + "commit": "30886de9b97d55c430a638480f14d16f4ba97d44",
43 + "dirty_tree": false
44 + },
45 + "thermal_level_at_collect": null
46 + },
47 + "config": {
48 + "per_domain": 1,
49 + "max_tokens": 32,
50 + "window": 32,
51 + "taus": "2.0",
52 + "modes": "verify-all"
53 + },
54 + "models": {
55 + "base": "mlx-community/Qwen3-32B-4bit",
56 + "verify": "mlx-community/Qwen3-32B-8bit",
57 + "judge": "mlx-community/Qwen3-8B-bf16"
58 + },
59 + "q8_streamed_bytes": 34810574466,
60 + "baseline_pure_q4_tokens_per_s": 23.983917284463768,
61 + "runs": [
62 + {
63 + "mode": "verify-all",
64 + "tau": 2.0,
65 + "window": 32,
66 + "tokens_per_s": 2.4801166056852075,
67 + "deferral_rate": 0.2760416666666667,
68 + "rollback_rate": 0.046875,
69 + "sweep_latency_s_mean": 3.026262025066535,
70 + "logical_verify_bytes_per_token": 2719576130.15625,
71 + "raw": {
72 + "tokens": 192,
73 + "deferred": 53,
74 + "sweeps": 15,
75 + "rollbacks": 9,
76 + "sweep_s": 45.393930375998025,
77 + "gen_s": 32.02178449800249,
78 + "logical_bytes": 522158616990
79 + }
80 + }
81 + ],
82 + "quality_8b_judge": {
83 + "pure_q4": {
84 + "mean_logprob_8b_judge": -0.3015170494715373,
85 + "n": 6
86 + },
87 + "verify-all_tau2.0": {
88 + "mean_logprob_8b_judge": -0.2445124089717865,
89 + "n": 6
90 + }
91 + }
92 +}
\ No newline at end of file
added results/candidate_01_scale32b/20260812T055506Z/results.json +115 −0
@@ -0,0 +1,115 @@
1 +{
2 + "experiment": "candidate_01_scale32b",
3 + "author": "Simon-Pierre Boucher",
4 + "contact": "contact@spboucher.ai",
5 + "manifest": {
6 + "author": "Simon-Pierre Boucher",
7 + "contact": "contact@spboucher.ai",
8 + "project": "localvm-research",
9 + "collected_utc": "2026-08-12T05:55:06.144401+00:00",
10 + "chip": {
11 + "brand": "Apple M5 Max",
12 + "arch": "arm64",
13 + "cores_total": 18,
14 + "cores_performance": 6,
15 + "cores_efficiency": 12,
16 + "gpu_cores": 40
17 + },
18 + "memory": {
19 + "unified_bytes": 51539607552,
20 + "unified_gb": 48.0,
21 + "pagesize": 16384
22 + },
23 + "ssd": {
24 + "model": "APPLE SSD AP2048Z",
25 + "size": "2 TB",
26 + "smart_status": "Verified"
27 + },
28 + "os": {
29 + "product": "macOS",
30 + "version": "27.0",
31 + "build": "26A5388g",
32 + "kernel": "27.0.0"
33 + },
34 + "software": {
35 + "python": "3.14.4",
36 + "mlx": "0.32.0",
37 + "mlx_lm": "0.31.3",
38 + "torch": null,
39 + "numpy": "2.5.2"
40 + },
41 + "git": {
42 + "commit": "30886de9b97d55c430a638480f14d16f4ba97d44",
43 + "dirty_tree": true
44 + },
45 + "thermal_level_at_collect": null
46 + },
47 + "config": {
48 + "per_domain": 2,
49 + "max_tokens": 96,
50 + "window": 32,
51 + "taus": "2.0",
52 + "modes": "margin,verify-all"
53 + },
54 + "models": {
55 + "base": "mlx-community/Qwen3-32B-4bit",
56 + "verify": "mlx-community/Qwen3-32B-8bit",
57 + "judge": "mlx-community/Qwen3-8B-bf16"
58 + },
59 + "q8_streamed_bytes": 34810574466,
60 + "baseline_pure_q4_tokens_per_s": 26.391281282242097,
61 + "runs": [
62 + {
63 + "mode": "margin",
64 + "tau": 2.0,
65 + "window": 32,
66 + "tokens_per_s": 1.6161952412668676,
67 + "deferral_rate": 1.125,
68 + "rollback_rate": 0.09722222222222222,
69 + "sweep_latency_s_mean": 2.9656049297462093,
70 + "logical_verify_bytes_per_token": 3928276632.4479165,
71 + "raw": {
72 + "tokens": 1152,
73 + "deferred": 1296,
74 + "sweeps": 130,
75 + "rollbacks": 112,
76 + "sweep_s": 385.5286408670072,
77 + "gen_s": 327.2565292569907,
78 + "logical_bytes": 4525374680580
79 + }
80 + },
81 + {
82 + "mode": "verify-all",
83 + "tau": 2.0,
84 + "window": 32,
85 + "tokens_per_s": 1.6903989953797323,
86 + "deferral_rate": 1.0234375,
87 + "rollback_rate": 0.0920138888888889,
88 + "sweep_latency_s_mean": 2.967401378024411,
89 + "logical_verify_bytes_per_token": 3716754044.546875,
90 + "raw": {
91 + "tokens": 1152,
92 + "deferred": 1179,
93 + "sweeps": 123,
94 + "rollbacks": 106,
95 + "sweep_s": 364.9903694970026,
96 + "gen_s": 316.50553954500083,
97 + "logical_bytes": 4281700659318
98 + }
99 + }
100 + ],
101 + "quality_8b_judge": {
102 + "pure_q4": {
103 + "mean_logprob_8b_judge": -0.7506343560914198,
104 + "n": 12
105 + },
106 + "margin_tau2.0": {
107 + "mean_logprob_8b_judge": -0.5459523188571135,
108 + "n": 12
109 + },
110 + "verify-all_tau2.0": {
111 + "mean_logprob_8b_judge": -0.46666722496350604,
112 + "n": 12
113 + }
114 + }
115 +}
\ No newline at end of file
116