SPB Git

spb/localvm-research Public License

Running LLMs larger than memory on a consumer Mac — falsification-driven research: margin-gated deferred refinement, out-of-core verification on Apple Silicon. TR-01 published.

Python 63.2% JavaScript 23.5% CSS 11.8% Shell 0.9% Makefile 0.5%

expA+expB: FFN block-energy trace campaign (concentration + temporal stability)

Shared instrumented pass: down_proj-input block energies per token/layer;
expA measures concentration vs uniform null; expB measures Jaccard decay,
union working sets, random null, and expC-lite domain locality.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simon-Pierre Boucher committed 5 h ago (Aug 12, 2026) parent c574d27

Showing 4 changed files with +351 and −38

modified experiments/micro/expA_weight_concentration/benchmark.py +162 −11
@@ -1,32 +1,183 @@
1 +#!/usr/bin/env python3
1 2 # =============================================================================
2 3 # Project : localvm-research
3 4 # File : experiments/micro/expA_weight_concentration/benchmark.py
4 −# Purpose : Benchmark runner: Weight contribution concentration: can a subset of weight blocks reproduce most of each layer's output?
5 +# Purpose : Per-token FFN block-energy concentration (trace shared with expB)
5 6 # Author : Simon-Pierre Boucher
6 7 # Contact : contact@spboucher.ai
7 −# Created : 2026-08-11
8 −# Modified : 2026-08-11
9 −# Platform : macOS / Apple Silicon (arm64)
8 +# Created : 2026-08-12
9 +# Modified : 2026-08-12
10 +# Platform : macOS / Apple Silicon (arm64) — MLX / Metal
10 11 # License : All rights reserved (research code)
11 12 # =============================================================================
13 +"""Experiment A — weight contribution concentration (charter §9.A).
12 14
13 −"""Benchmark entry point for expA_weight_concentration.
15 +Records SwiGLU intermediate-activation energy per 64-neuron block, per token,
16 +per layer, on the bf16 reference model. Writes concentration aggregates to
17 +results.json and the raw block-energy trace (npz) for expB.
14 18
15 −Must embed the hardware manifest in all result output
16 −(see benchmarks/hardware_manifest.py) and write results to
17 −results/expA_weight_concentration/<timestamp>/.
19 +Usage:
20 + .venv/bin/python benchmark.py [--per-domain 8] [--gen-tokens 128] [--block 64]
18 21 """
19 22
23 +from __future__ import annotations
24 +
25 +import argparse
26 +import json
20 27 import sys
28 +import time
29 +from datetime import datetime, timezone
21 30 from pathlib import Path
22 31
23 −sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "benchmarks"))
32 +import mlx.core as mx
33 +import mlx.nn as nn
34 +import numpy as np
35 +from mlx_lm import load
36 +
37 +REPO_ROOT = Path(__file__).resolve().parents[3]
38 +sys.path.insert(0, str(REPO_ROOT / "benchmarks"))
39 +sys.path.insert(0, str(REPO_ROOT / "src"))
24 40 from hardware_manifest import collect_manifest # noqa: E402
41 +from localvm.quality.decision_stats import greedy_generate # noqa: E402
42 +
43 +
44 +class DownProjRecorder(nn.Module):
45 + """Wraps a down_proj Linear; records block energy of its input (the SwiGLU
46 + intermediate activation) for positions in `window`."""
47 +
48 + def __init__(self, inner: nn.Module, block: int):
49 + super().__init__()
50 + self.inner = inner
51 + self.block = block
52 + self.window: tuple[int, int] | None = None
53 + self.block_energy: np.ndarray | None = None
54 + self.neuron_energy: np.ndarray | None = None
55 +
56 + def __call__(self, x):
57 + if self.window is not None:
58 + a, b = self.window
59 + h = x[0, a:b].astype(mx.float32)
60 + e = mx.square(h)
61 + be = e.reshape(h.shape[0], -1, self.block).sum(axis=-1)
62 + mx.eval(be)
63 + self.block_energy = np.array(be)
64 + ne = np.array(e) # (T, D_int) — reduced immediately by caller
65 + self.neuron_energy = ne
66 + return self.inner(x)
67 +
68 +
69 +def concentration_stats(energy: np.ndarray, fracs=(0.1, 0.2, 0.4, 0.6),
70 + targets=(0.90, 0.95, 0.99)) -> dict:
71 + """energy: (T, N). Returns mean energy captured by top-f fraction and mean
72 + fraction of units needed to reach target energy."""
73 + T, N = energy.shape
74 + srt = np.sort(energy, axis=1)[:, ::-1]
75 + csum = np.cumsum(srt, axis=1)
76 + total = csum[:, -1:] + 1e-12
77 + frac_captured = {}
78 + for f in fracs:
79 + k = max(1, int(round(f * N)))
80 + frac_captured[f] = float(np.mean(csum[:, k - 1] / total[:, 0]))
81 + needed = {}
82 + ratio = csum / total
83 + for t in targets:
84 + idx = np.argmax(ratio >= t, axis=1) + 1
85 + needed[t] = float(np.mean(idx / N))
86 + return {"top_frac_energy": frac_captured, "frac_needed_for": needed}
25 87
26 88
27 89 def main() -> None:
28 − manifest = collect_manifest()
29 − raise NotImplementedError("experiment not yet implemented")
90 + ap = argparse.ArgumentParser()
91 + ap.add_argument("--model", default="mlx-community/Qwen3-1.7B-bf16")
92 + ap.add_argument("--gen-tokens", type=int, default=128)
93 + ap.add_argument("--per-domain", type=int, default=8)
94 + ap.add_argument("--block", type=int, default=64)
95 + args = ap.parse_args()
96 +
97 + domains = json.loads((REPO_ROOT / "benchmarks/datasets/eval_prompts.json").read_text())["domains"]
98 + print(f"loading {args.model} …", flush=True)
99 + model, tokenizer = load(args.model)
100 + layers = model.model.layers
101 + n_layers = len(layers)
102 +
103 + recorders = []
104 + for layer in layers:
105 + rec = DownProjRecorder(layer.mlp.down_proj, args.block)
106 + layer.mlp.down_proj = rec
107 + recorders.append(rec)
108 +
109 + trajectories = []
110 + t0 = time.time()
111 + for domain, plist in domains.items():
112 + for prompt in plist[: args.per_domain]:
113 + ids = tokenizer.apply_chat_template(
114 + [{"role": "user", "content": prompt}], add_generation_prompt=True)
115 + for r in recorders:
116 + r.window = None # no recording during generation
117 + gen = greedy_generate(model, tokenizer, ids, args.gen_tokens)
118 + if len(gen) >= 8:
119 + trajectories.append({"domain": domain, "full_ids": list(ids) + gen, "start": len(ids)})
120 + print(f"{len(trajectories)} trajectories in {time.time()-t0:.0f}s", flush=True)
121 +
122 + block_traces = [] # per traj: (T, n_layers, n_blocks) f16
123 + neuron_stats = [] # per traj per layer concentration dicts
124 + index = []
125 + for ti, t in enumerate(trajectories):
126 + a, b = t["start"] - 1, len(t["full_ids"]) - 1
127 + for r in recorders:
128 + r.window = (a, b)
129 + model(mx.array(t["full_ids"])[None])
130 + per_layer_blocks = np.stack([r.block_energy for r in recorders], axis=1) # (T, L, B)
131 + block_traces.append(per_layer_blocks.astype(np.float16))
132 + neuron_stats.append([concentration_stats(r.neuron_energy) for r in recorders])
133 + for r in recorders:
134 + r.neuron_energy = None
135 + index.append({"traj": ti, "domain": t["domain"], "n_pos": b - a})
136 + if (ti + 1) % 12 == 0:
137 + print(f" traced {ti+1}/{len(trajectories)}", flush=True)
138 +
139 + all_blocks = np.concatenate(block_traces, axis=0) # (P, L, B)
140 + P, L, B = all_blocks.shape
141 + print(f"trace shape {all_blocks.shape}", flush=True)
142 +
143 + # expA aggregates at block granularity
144 + per_layer = [concentration_stats(all_blocks[:, li, :].astype(np.float32)) for li in range(L)]
145 + overall = concentration_stats(all_blocks.reshape(P * L, B).astype(np.float32))
146 + per_domain = {}
147 + pos_domain = np.concatenate([[ix["domain"]] * ix["n_pos"] for ix in index])
148 + for dom in domains:
149 + sel = all_blocks[pos_domain == dom]
150 + per_domain[dom] = concentration_stats(sel.reshape(-1, B).astype(np.float32))
151 + # neuron-granularity mean across trajectories/layers
152 + neuron_overall = {
153 + "top_frac_energy": {f: float(np.mean([s["top_frac_energy"][f] for ns in neuron_stats for s in ns]))
154 + for f in (0.1, 0.2, 0.4, 0.6)},
155 + "frac_needed_for": {t: float(np.mean([s["frac_needed_for"][t] for ns in neuron_stats for s in ns]))
156 + for t in (0.90, 0.95, 0.99)},
157 + }
158 +
159 + ts = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
160 + out_dir = REPO_ROOT / "results" / "expA_weight_concentration" / ts
161 + out_dir.mkdir(parents=True)
162 + np.savez_compressed(out_dir / "block_energy_trace.npz",
163 + blocks=all_blocks,
164 + domains=pos_domain,
165 + traj_id=np.concatenate([[ix["traj"]] * ix["n_pos"] for ix in index]))
166 + (out_dir / "results.json").write_text(json.dumps({
167 + "experiment": "expA_weight_concentration",
168 + "author": "Simon-Pierre Boucher",
169 + "contact": "contact@spboucher.ai",
170 + "manifest": collect_manifest(),
171 + "config": vars(args),
172 + "n_positions": int(P), "n_layers": int(L), "n_blocks": int(B),
173 + "block_granularity": {"overall": overall,
174 + "per_layer": {str(i): s for i, s in enumerate(per_layer)},
175 + "per_domain": per_domain},
176 + "neuron_granularity": neuron_overall,
177 + }, indent=2, default=float))
178 + print(f"\nwrote {out_dir}/results.json (+ block_energy_trace.npz for expB)")
179 + print("overall block-64:", json.dumps(overall, default=float))
180 + print("neuron-level :", json.dumps(neuron_overall, default=float))
30 181
31 182
32 183 if __name__ == "__main__":
modified experiments/micro/expA_weight_concentration/hypothesis.md +33 −8
@@ -3,31 +3,56 @@ project: localvm-research
3 3 document: expA_weight_concentration/hypothesis
4 4 author: Simon-Pierre Boucher
5 5 contact: contact@spboucher.ai
6 −created: 2026-08-11
6 +created: 2026-08-12
7 7 status: draft
8 8 ---
9 9
10 10 # Hypothesis — expA_weight_concentration
11 11
12 +After expF killed layer-granularity escalation, this measures the next grain
13 +down: are FFN weight *blocks* (bundled neurons) unequally important per token?
14 +Feeds G01/G06/G07 and expE (partial GEMM); shares its trace with expB.
15 +
12 16 ```text
13 17 Hypothesis
14 − <what we believe and why>
18 + Per-token FFN intermediate-activation energy is concentrated: on a modern
19 + SwiGLU model, the top 20% of 64-neuron blocks capture ≥60% of the energy,
20 + and ≤50% of blocks suffice for 95% of the energy (per token, averaged
21 + across positions and domains). Per-neuron concentration is substantially
22 + stronger than block-64 concentration (bundling cost is real but moderate).
15 23
16 24 Falsification criterion
17 − <the concrete measurable outcome that would prove this wrong>
25 + If capturing 95% of per-token energy requires >70% of 64-neuron blocks
26 + (near-uniform importance), then block-level weight selection cannot cut
27 + bytes materially on this architecture and G06-style paging must rely on
28 + thresholded sparsity of individual neurons or die; C1 escalation-byte
29 + reduction via block selection (route c from expF) is dead too.
18 30
19 31 Method
20 − <exact procedure, model(s), data, seeds, measurement points>
32 + Qwen3-1.7B bf16. Wrap every layer's mlp.down_proj with a recorder; its
33 + input IS the SwiGLU intermediate activation h = silu(gate(x))·up(x),
34 + whose per-neuron magnitude determines the contribution of up/gate rows
35 + and down columns (the Gate-Up-Down bundle of the paging literature).
36 + Forward the 48 reference trajectories (same protocol as expG/D/F,
37 + greedy 128-token continuations, teacher-forced positions only).
38 + Record per predicted position: block energy (sum of h² over 64-neuron
39 + blocks; 96 blocks × 28 layers), stored float16 npz for expB reuse; plus
40 + streaming per-neuron stats (fraction of neurons for 90/95/99% energy).
41 + Report: energy captured by top {10,20,40,60}% blocks; blocks needed for
42 + {90,95,99}% energy; per-layer, per-domain aggregates; neuron-vs-block
43 + comparison.
21 44
22 45 Baseline
23 − <what this is compared against — no straw men>
46 + Uniform importance (top k% of blocks capture exactly k% of energy) —
47 + the null hypothesis; and per-neuron granularity as the upper bound on
48 + achievable concentration.
24 49
25 50 Result
26 − <filled after the run: numbers, with mean/median/std and run count>
51 + <filled after the run>
27 52
28 53 Interpretation
29 − <what the numbers mean; alternative explanations considered>
54 + <filled after the run>
30 55
31 56 Next experiment
32 − <the most informative follow-up given this result>
57 + <filled after the run>
33 58 ```
modified experiments/micro/expB_token_stability/benchmark.py +127 −11
@@ -1,32 +1,148 @@
1 +#!/usr/bin/env python3
1 2 # =============================================================================
2 3 # Project : localvm-research
3 4 # File : experiments/micro/expB_token_stability/benchmark.py
4 −# Purpose : Benchmark runner: Stability of important weight blocks across consecutive tokens (Jaccard, transitions, working-set lifetime)
5 +# Purpose : Temporal stability of per-token important FFN block sets
6 +# (consumes expA's block-energy trace)
5 7 # Author : Simon-Pierre Boucher
6 8 # Contact : contact@spboucher.ai
7 −# Created : 2026-08-11
8 −# Modified : 2026-08-11
9 +# Created : 2026-08-12
10 +# Modified : 2026-08-12
9 11 # Platform : macOS / Apple Silicon (arm64)
10 12 # License : All rights reserved (research code)
11 13 # =============================================================================
14 +"""Experiment B — stability across consecutive tokens (charter §9.B).
12 15
13 −"""Benchmark entry point for expB_token_stability.
14 −
15 −Must embed the hardware manifest in all result output
16 −(see benchmarks/hardware_manifest.py) and write results to
17 −results/expB_token_stability/<timestamp>/.
16 +Usage:
17 + .venv/bin/python benchmark.py [--trace <path/to/block_energy_trace.npz>]
18 + [--target 0.95]
18 19 """
19 20
21 +from __future__ import annotations
22 +
23 +import argparse
24 +import glob
25 +import json
20 26 import sys
27 +from datetime import datetime, timezone
21 28 from pathlib import Path
22 29
23 −sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "benchmarks"))
30 +import numpy as np
31 +
32 +REPO_ROOT = Path(__file__).resolve().parents[3]
33 +sys.path.insert(0, str(REPO_ROOT / "benchmarks"))
24 34 from hardware_manifest import collect_manifest # noqa: E402
25 35
36 +DELTAS = [1, 2, 4, 8, 16, 32]
37 +WINDOWS = [8, 32, 128]
38 +
39 +
40 +def top_sets(energy: np.ndarray, target: float) -> list[np.ndarray]:
41 + """energy (T, B) → per-position boolean mask of the smallest block set
42 + covering `target` of the energy."""
43 + T, B = energy.shape
44 + order = np.argsort(energy, axis=1)[:, ::-1]
45 + srt = np.take_along_axis(energy, order, axis=1)
46 + csum = np.cumsum(srt, axis=1)
47 + total = csum[:, -1:] + 1e-12
48 + kneed = np.argmax(csum / total >= target, axis=1) + 1
49 + masks = np.zeros((T, B), dtype=bool)
50 + for t in range(T):
51 + masks[t, order[t, : kneed[t]]] = True
52 + return masks
53 +
54 +
55 +def jaccard(a: np.ndarray, b: np.ndarray) -> float:
56 + inter = np.logical_and(a, b).sum()
57 + union = np.logical_or(a, b).sum()
58 + return float(inter / union) if union else 1.0
59 +
26 60
27 61 def main() -> None:
28 − manifest = collect_manifest()
29 − raise NotImplementedError("experiment not yet implemented")
62 + ap = argparse.ArgumentParser()
63 + ap.add_argument("--trace", default=None)
64 + ap.add_argument("--target", type=float, default=0.95)
65 + args = ap.parse_args()
66 + trace_path = args.trace or sorted(
67 + glob.glob(str(REPO_ROOT / "results/expA_weight_concentration/*/block_energy_trace.npz")))[-1]
68 + z = np.load(trace_path, allow_pickle=False)
69 + blocks = z["blocks"].astype(np.float32) # (P, L, B)
70 + domains = z["domains"]
71 + traj_id = z["traj_id"]
72 + P, L, B = blocks.shape
73 + print(f"trace {trace_path}: {blocks.shape}", flush=True)
74 +
75 + jac = {d: [] for d in DELTAS}
76 + jac_random = []
77 + union_frac = {w: [] for w in WINDOWS}
78 + per_layer_j1 = [[] for _ in range(L)]
79 + traj_masks_union = {} # (traj, layer) -> aggregate mask, for expC-lite
80 + traj_domain = {}
81 +
82 + rng = np.random.default_rng(7)
83 + for ti in np.unique(traj_id):
84 + sel = traj_id == ti
85 + traj_domain[int(ti)] = str(domains[sel][0])
86 + for li in range(L):
87 + masks = top_sets(blocks[sel, li, :], args.target)
88 + T = masks.shape[0]
89 + for d in DELTAS:
90 + if T > d:
91 + vals = [jaccard(masks[t], masks[t + d]) for t in range(T - d)]
92 + jac[d].extend(vals)
93 + if d == 1:
94 + per_layer_j1[li].extend(vals)
95 + # random-set null at matched sizes (δ=1 pairs)
96 + sizes = masks.sum(axis=1)
97 + for t in range(min(T - 1, 8)):
98 + a = np.zeros(B, bool); a[rng.choice(B, sizes[t], replace=False)] = True
99 + b = np.zeros(B, bool); b[rng.choice(B, sizes[t + 1], replace=False)] = True
100 + jac_random.append(jaccard(a, b))
101 + for w in WINDOWS:
102 + for s in range(0, T - w + 1, w):
103 + union_frac[w].append(float(masks[s:s + w].any(axis=0).mean()))
104 + traj_masks_union[(int(ti), li)] = masks.any(axis=0)
105 +
106 + # expC-lite: within- vs across-domain overlap of per-trajectory unions
107 + tids = sorted(traj_domain)
108 + within, across = [], []
109 + for i in range(len(tids)):
110 + for j in range(i + 1, len(tids)):
111 + v = np.mean([jaccard(traj_masks_union[(tids[i], li)], traj_masks_union[(tids[j], li)])
112 + for li in range(0, L, 4)])
113 + (within if traj_domain[tids[i]] == traj_domain[tids[j]] else across).append(v)
114 +
115 + result = {
116 + "jaccard_by_delta": {str(d): {"mean": float(np.mean(v)), "std": float(np.std(v)), "n": len(v)}
117 + for d, v in jac.items()},
118 + "jaccard_random_null": {"mean": float(np.mean(jac_random)), "std": float(np.std(jac_random))},
119 + "union_working_set_frac_by_window": {str(w): {"mean": float(np.mean(v)), "std": float(np.std(v))}
120 + for w, v in union_frac.items()},
121 + "per_layer_jaccard1_mean": [float(np.mean(v)) for v in per_layer_j1],
122 + "expC_lite_domain_locality": {
123 + "within_domain_union_jaccard": float(np.mean(within)),
124 + "across_domain_union_jaccard": float(np.mean(across)),
125 + },
126 + "mean_topset_frac": float(np.mean([m.mean() for m in
127 + [top_sets(blocks[traj_id == t, li, :], args.target)
128 + for t in np.unique(traj_id)[:4] for li in (0, L // 2, L - 1)]])),
129 + }
130 + ts = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
131 + out_dir = REPO_ROOT / "results" / "expB_token_stability" / ts
132 + out_dir.mkdir(parents=True)
133 + (out_dir / "results.json").write_text(json.dumps({
134 + "experiment": "expB_token_stability",
135 + "author": "Simon-Pierre Boucher",
136 + "contact": "contact@spboucher.ai",
137 + "manifest": collect_manifest(),
138 + "config": {"trace": str(trace_path), "target": args.target},
139 + **result,
140 + }, indent=2))
141 + print(json.dumps(result["jaccard_by_delta"], indent=1))
142 + print("random null:", result["jaccard_random_null"])
143 + print("union by window:", json.dumps(result["union_working_set_frac_by_window"]))
144 + print("domain locality:", result["expC_lite_domain_locality"])
145 + print(f"\nwrote {out_dir / 'results.json'}")
30 146
31 147
32 148 if __name__ == "__main__":
modified experiments/micro/expB_token_stability/hypothesis.md +29 −8
@@ -3,31 +3,52 @@ project: localvm-research
3 3 document: expB_token_stability/hypothesis
4 4 author: Simon-Pierre Boucher
5 5 contact: contact@spboucher.ai
6 −created: 2026-08-11
6 +created: 2026-08-12
7 7 status: draft
8 8 ---
9 9
10 10 # Hypothesis — expB_token_stability
11 11
12 +Consumes expA's block-energy trace. Decides the temporal-locality route for
13 +C1's escalation bytes (and G01/G06/G12/G13 cache designs): is the important-
14 +block set stable enough across consecutive tokens to cache?
15 +
12 16 ```text
13 17 Hypothesis
14 − <what we believe and why>
18 + The per-token important-block set (blocks covering 95% of FFN energy) is
19 + temporally sticky: mean Jaccard(t, t+1) ≥ 0.5, decaying slowly with
20 + distance, and the UNION working set over a 128-token generation stays
21 + well below the full model (≤80% of blocks) — i.e., a generation has a
22 + reusable working set that a RAM cache can hold, so residual/expert bytes
23 + are fetched once per window, not once per token.
15 24
16 25 Falsification criterion
17 − <the concrete measurable outcome that would prove this wrong>
26 + If Jaccard(t, t+1) < 0.3 (set churns almost completely every token) or
27 + the 128-token union ≥ 95% of all blocks (no working set exists at
28 + generation scale), the temporal-locality route (b) for C1 dies, leaving
29 + only batch amortization — C1 then merges into C2 (amortized
30 + verification), and G06/G12 prefetchers must predict, not cache.
18 31
19 32 Method
20 − <exact procedure, model(s), data, seeds, measurement points>
33 + Load expA's block_energy_trace.npz (positions × 28 layers × 96 blocks,
34 + with domain and trajectory ids). Per trajectory and layer:
35 + top-set(t) = smallest block set covering 95% of position-t energy.
36 + Metrics: mean Jaccard(t, t+δ) for δ ∈ {1, 2, 4, 8, 16, 32}; union
37 + working-set fraction over windows {8, 32, 128} tokens; per-layer and
38 + per-domain aggregates. Cross-prompt preview (expC-lite): per-trajectory
39 + aggregate top-set, Jaccard within-domain vs across-domain.
21 40
22 41 Baseline
23 − <what this is compared against — no straw men>
42 + Random sets of identical size (expected Jaccard for independent draws)
43 + — the null; and δ→∞ behavior (unconditional overlap between distant
44 + tokens) as the floor stickiness must beat.
24 45
25 46 Result
26 − <filled after the run: numbers, with mean/median/std and run count>
47 + <filled after the run>
27 48
28 49 Interpretation
29 − <what the numbers mean; alternative explanations considered>
50 + <filled after the run>
30 51
31 52 Next experiment
32 − <the most informative follow-up given this result>
53 + <filled after the run>
33 54 ```
34 55