SPB Git

spb/localvm-research Public License

Running LLMs larger than memory on a consumer Mac — falsification-driven research: margin-gated deferred refinement, out-of-core verification on Apple Silicon. TR-01 published.

Python 63.2% JavaScript 23.5% CSS 11.8% Shell 0.9% Makefile 0.5%

expF: layer-sensitivity benchmark + hypothesis (degrade-one / repair-one / repair-top-k)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simon-Pierre Boucher committed 5 h ago (Aug 12, 2026) parent c089a80

Showing 2 changed files with +191 and −19

modified experiments/micro/expF_error_accumulation/benchmark.py +158 −11
@@ -1,32 +1,179 @@
1 +#!/usr/bin/env python3
1 2 # =============================================================================
2 3 # Project : localvm-research
3 4 # File : experiments/micro/expF_error_accumulation/benchmark.py
4 −# Purpose : Benchmark runner: Error accumulation: which layers tolerate, amplify, or recover from controlled approximation
5 +# Purpose : Layer-sensitivity map — degrade-one / repair-one / repair-top-k
6 +# (does layer-restricted escalation cut bytes-per-escalation?)
5 7 # Author : Simon-Pierre Boucher
6 8 # Contact : contact@spboucher.ai
7 −# Created : 2026-08-11
8 −# Modified : 2026-08-11
9 −# Platform : macOS / Apple Silicon (arm64)
9 +# Created : 2026-08-12
10 +# Modified : 2026-08-12
11 +# Platform : macOS / Apple Silicon (arm64) — MLX / Metal
10 12 # License : All rights reserved (research code)
11 13 # =============================================================================
14 +"""Experiment F — error accumulation / layer sensitivity (charter §9.F).
12 15
13 −"""Benchmark entry point for expF_error_accumulation.
16 +Per depth-group 4-bit degradation and repair on Qwen3-1.7B, teacher-forced
17 +over reference greedy trajectories.
14 18
15 −Must embed the hardware manifest in all result output
16 −(see benchmarks/hardware_manifest.py) and write results to
17 −results/expF_error_accumulation/<timestamp>/.
19 +Usage:
20 + .venv/bin/python benchmark.py [--per-domain 8] [--gen-tokens 128] [--groups 7]
18 21 """
19 22
23 +from __future__ import annotations
24 +
25 +import argparse
26 +import json
20 27 import sys
28 +import time
29 +from datetime import datetime, timezone
21 30 from pathlib import Path
22 31
23 −sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "benchmarks"))
32 +import mlx.core as mx
33 +import mlx.nn as nn
34 +import numpy as np
35 +from mlx_lm import load
36 +
37 +REPO_ROOT = Path(__file__).resolve().parents[3]
38 +sys.path.insert(0, str(REPO_ROOT / "benchmarks"))
39 +sys.path.insert(0, str(REPO_ROOT / "src"))
24 40 from hardware_manifest import collect_manifest # noqa: E402
41 +from localvm.quality.decision_stats import greedy_generate, kl_ref_vs, teacher_forced_stats # noqa: E402
42 +
43 +GROUP_SIZE = 64
44 +BITS = 4
45 +
46 +
47 +def layer_index(path: str) -> int | None:
48 + parts = path.split(".")
49 + for i, p in enumerate(parts):
50 + if p == "layers" and i + 1 < len(parts) and parts[i + 1].isdigit():
51 + return int(parts[i + 1])
52 + return None
53 +
54 +
55 +def quantized_weights_by_layer(model) -> dict[str, tuple[int, mx.array]]:
56 + """{param_path: (layer_idx, 4-bit-dequantized bf16 weight)} for all
57 + divisible Linear layers inside transformer blocks."""
58 + out = {}
59 + for path, module in model.named_modules():
60 + li = layer_index(path)
61 + if li is None or not isinstance(module, nn.Linear):
62 + continue
63 + if module.weight.shape[-1] % GROUP_SIZE != 0:
64 + continue
65 + w = module.weight.astype(mx.float32)
66 + qw, sc, bi = mx.quantize(w, group_size=GROUP_SIZE, bits=BITS)
67 + deq = mx.dequantize(qw, sc, bi, group_size=GROUP_SIZE, bits=BITS).astype(mx.bfloat16)
68 + mx.eval(deq)
69 + out[path] = (li, deq)
70 + return out
71 +
72 +
73 +def apply_config(model, qweights: dict, originals: dict, degrade_layers: set[int]) -> None:
74 + """Set each eligible Linear to 4-bit dequant if its layer ∈ degrade_layers,
75 + else restore the original bf16 weight."""
76 + for path, module in model.named_modules():
77 + if path in qweights:
78 + li, deq = qweights[path]
79 + module.weight = deq if li in degrade_layers else originals[path]
80 +
81 +
82 +def evaluate(model, trajectories, ref_stats) -> dict:
83 + agrees, kls = [], []
84 + for t, ref in zip(trajectories, ref_stats):
85 + qs = teacher_forced_stats(model, t["full_ids"], t["start"])
86 + ref_next = np.array(t["full_ids"][t["start"]:])
87 + agrees.append((qs["argmax"] == ref_next).astype(np.int8))
88 + kls.append(kl_ref_vs(qs["logprobs"], ref["logprobs"]))
89 + return {
90 + "agreement_rate": float(np.concatenate(agrees).mean()),
91 + "mean_kl": float(np.mean(np.concatenate(kls))),
92 + }
25 93
26 94
27 95 def main() -> None:
28 − manifest = collect_manifest()
29 − raise NotImplementedError("experiment not yet implemented")
96 + ap = argparse.ArgumentParser()
97 + ap.add_argument("--model", default="mlx-community/Qwen3-1.7B-bf16")
98 + ap.add_argument("--gen-tokens", type=int, default=128)
99 + ap.add_argument("--per-domain", type=int, default=8)
100 + ap.add_argument("--groups", type=int, default=7)
101 + args = ap.parse_args()
102 +
103 + domains = json.loads((REPO_ROOT / "benchmarks/datasets/eval_prompts.json").read_text())["domains"]
104 + print(f"loading {args.model} …", flush=True)
105 + model, tokenizer = load(args.model)
106 + n_layers = len(model.model.layers)
107 + bounds = np.linspace(0, n_layers, args.groups + 1).astype(int)
108 + groups = [set(range(bounds[i], bounds[i + 1])) for i in range(args.groups)]
109 +
110 + trajectories = []
111 + t0 = time.time()
112 + for domain, plist in domains.items():
113 + for prompt in plist[: args.per_domain]:
114 + ids = tokenizer.apply_chat_template(
115 + [{"role": "user", "content": prompt}], add_generation_prompt=True)
116 + gen = greedy_generate(model, tokenizer, ids, args.gen_tokens)
117 + if len(gen) >= 8:
118 + trajectories.append({"domain": domain, "full_ids": list(ids) + gen, "start": len(ids)})
119 + print(f"{len(trajectories)} trajectories in {time.time()-t0:.0f}s", flush=True)
120 + ref_stats = [teacher_forced_stats(model, t["full_ids"], t["start"]) for t in trajectories]
121 +
122 + print("precomputing 4-bit weights …", flush=True)
123 + qweights = quantized_weights_by_layer(model)
124 + originals = {p: m.weight for p, m in model.named_modules() if p in qweights}
125 + all_layers = set(range(n_layers))
126 +
127 + runs: dict[str, dict] = {}
128 +
129 + def run(tag: str, degrade: set[int]) -> dict:
130 + apply_config(model, qweights, originals, degrade)
131 + r = evaluate(model, trajectories, ref_stats)
132 + r["degraded_layers"] = sorted(degrade)
133 + runs[tag] = r
134 + print(f" {tag:>24}: agree={r['agreement_rate']:.4f} KL={r['mean_kl']:.4f}", flush=True)
135 + return r
136 +
137 + print("all-4-bit floor:", flush=True)
138 + floor = run("all_4bit", all_layers)
139 + print("degrade-one (rest bf16):", flush=True)
140 + for gi, g in enumerate(groups):
141 + run(f"degrade_g{gi}_L{min(g)}-{max(g)}", g)
142 + print("repair-one (rest 4-bit):", flush=True)
143 + for gi, g in enumerate(groups):
144 + run(f"repair_g{gi}_L{min(g)}-{max(g)}", all_layers - g)
145 +
146 + # repair-top-k by measured repair value
147 + lost = 1.0 - floor["agreement_rate"]
148 + repair_value = {
149 + gi: runs[f"repair_g{gi}_L{min(g)}-{max(g)}"]["agreement_rate"] - floor["agreement_rate"]
150 + for gi, g in enumerate(groups)
151 + }
152 + order = sorted(repair_value, key=repair_value.get, reverse=True)
153 + print("repair-top-k (best groups bf16):", flush=True)
154 + for k in (2, 3):
155 + keep = set().union(*(groups[gi] for gi in order[:k]))
156 + run(f"repair_top{k}_groups_{sorted(order[:k])}", all_layers - keep)
157 +
158 + apply_config(model, qweights, originals, set()) # restore
159 +
160 + ts = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
161 + out_dir = REPO_ROOT / "results" / "expF_error_accumulation" / ts
162 + out_dir.mkdir(parents=True)
163 + (out_dir / "results.json").write_text(json.dumps({
164 + "experiment": "expF_error_accumulation",
165 + "author": "Simon-Pierre Boucher",
166 + "contact": "contact@spboucher.ai",
167 + "manifest": collect_manifest(),
168 + "config": vars(args),
169 + "bits": BITS, "group_size": GROUP_SIZE,
170 + "n_layers": n_layers,
171 + "layer_groups": [sorted(g) for g in groups],
172 + "agreement_lost_all4bit": lost,
173 + "repair_value_by_group": {str(k): v for k, v in repair_value.items()},
174 + "runs": runs,
175 + }, indent=2))
176 + print(f"\nwrote {out_dir / 'results.json'}")
30 177
31 178
32 179 if __name__ == "__main__":
modified experiments/micro/expF_error_accumulation/hypothesis.md +33 −8
@@ -3,31 +3,56 @@ project: localvm-research
3 3 document: expF_error_accumulation/hypothesis
4 4 author: Simon-Pierre Boucher
5 5 contact: contact@spboucher.ai
6 −created: 2026-08-11
6 +created: 2026-08-12
7 7 status: draft
8 8 ---
9 9
10 10 # Hypothesis — expF_error_accumulation
11 11
12 +Follows expD: escalation currently means touching the FULL residual; if layer
13 +sensitivity is concentrated, escalation can be restricted to a subset of
14 +layers and candidate C1's bytes-per-escalation drops proportionally.
15 +
12 16 ```text
13 17 Hypothesis
14 − <what we believe and why>
18 + Layer sensitivity to quantization error is strongly non-uniform: degrading
19 + a single depth-group to 4-bit (rest bf16) hurts agreement unevenly across
20 + groups (≥3× spread between most and least sensitive), and symmetrically,
21 + repairing only the most sensitive ~25% of layers (bf16 in-group, 4-bit
22 + elsewhere) recovers a disproportionate share — ≥40% — of the agreement
23 + lost by the all-4-bit model.
15 24
16 25 Falsification criterion
17 − <the concrete measurable outcome that would prove this wrong>
26 + If per-group degradation effects are near-uniform (<2× spread), or if
27 + repairing the best 25% of layers recovers <20% of the lost agreement
28 + (i.e., error is diffuse and cooperative across depth), then
29 + layer-restricted escalation cannot cut bytes-per-escalation materially
30 + and C1 must rely entirely on temporal locality (expB) or block-level
31 + selection (expE).
18 32
19 33 Method
20 − <exact procedure, model(s), data, seeds, measurement points>
34 + Qwen3-1.7B bf16 reference, same 48-trajectory teacher-forced protocol
35 + (benchmarks/datasets/eval_prompts.json, 128 tokens, greedy reference).
36 + 28 transformer layers → 7 contiguous depth groups of 4.
37 + (i) DEGRADE-ONE: quantize (affine g64, 4-bit) all divisible Linear
38 + layers of one group; rest bf16. 7 runs → sensitivity map.
39 + (ii) REPAIR-ONE: all layers 4-bit except one group at bf16. 7 runs →
40 + repair-value map, plus the all-4-bit floor (from expD stage0 B).
41 + (iii) REPAIR-TOP-K: bf16 for the k most-repairing groups (k=1,2), 4-bit
42 + elsewhere → cumulative repair curve vs bytes.
43 + Metrics per config: agreement with reference, mean KL. Embedding/head
44 + layers excluded (kept bf16 throughout, as in expD/expG).
21 45
22 46 Baseline
23 − <what this is compared against — no straw men>
47 + All-bf16 (agreement=1 by construction) and all-4-bit (87.5%, expD
48 + stage0 B) bracket every configuration. No straw men.
24 49
25 50 Result
26 − <filled after the run: numbers, with mean/median/std and run count>
51 + <filled after the run>
27 52
28 53 Interpretation
29 − <what the numbers mean; alternative explanations considered>
54 + <filled after the run>
30 55
31 56 Next experiment
32 − <the most informative follow-up given this result>
57 + <filled after the run>
33 58 ```
34 59