SPB Git

spb/localvm-research Public License

Running LLMs larger than memory on a consumer Mac — falsification-driven research: margin-gated deferred refinement, out-of-core verification on Apple Silicon. TR-01 published.

Python 63.2% JavaScript 23.5% CSS 11.8% Shell 0.9% Makefile 0.5%

expG: decision-stability benchmark + hypothesis + cross-domain eval prompts

Gate-Zero measurement: reference greedy trajectories (Qwen3-1.7B bf16),
teacher-forced 2/3/4/8-bit MLX variants, joint margin×agreement matrix,
AUROC, escalation curves. Kill: AUROC<0.65 or >50% escalation at 99%.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simon-Pierre Boucher committed 4 h ago (Aug 12, 2026) parent f1981e4

Showing 3 changed files with +315 and −19

added benchmarks/datasets/eval_prompts.json +68 −0
@@ -0,0 +1,68 @@
1 +{
2 + "author": "Simon-Pierre Boucher",
3 + "contact": "contact@spboucher.ai",
4 + "project": "localvm-research",
5 + "description": "Cross-domain eval prompts (charter section 8.2/9): code, math, chat, French, English prose, reasoning. Used for teacher-forced decision-stability and importance-trace experiments.",
6 + "domains": {
7 + "code": [
8 + "Write a Python function that merges two sorted lists into one sorted list without using sort().",
9 + "Explain what this C snippet does: `while (*d++ = *s++);` and why it is dangerous.",
10 + "Implement a least-recently-used cache in Python using collections.OrderedDict.",
11 + "Write a SQL query returning the three highest-revenue customers per region.",
12 + "Refactor this loop into a list comprehension: result = []\nfor x in items:\n if x > 0:\n result.append(x * 2)",
13 + "Write a shell one-liner that finds the 10 largest files under /var/log.",
14 + "Explain the difference between a mutex and a semaphore with a short code example.",
15 + "Write a JavaScript debounce function and explain when to use it."
16 + ],
17 + "math": [
18 + "Compute the derivative of f(x) = x^3 * ln(x) and simplify.",
19 + "A train travels 240 km in 3 hours, then 180 km in 2 hours. What is its average speed for the whole trip?",
20 + "Prove that the sum of two odd integers is even.",
21 + "Solve the system: 2x + 3y = 12 and x - y = 1.",
22 + "What is the probability of getting exactly two heads in four fair coin flips?",
23 + "Factor the polynomial x^2 - 5x + 6 and verify the roots.",
24 + "A rectangle's perimeter is 36 and its length is twice its width. Find its area.",
25 + "Explain why the harmonic series diverges."
26 + ],
27 + "chat": [
28 + "I'm feeling overwhelmed at work lately. Any advice on how to handle it?",
29 + "What's a good gift idea for a friend who loves cooking?",
30 + "Can you help me write a polite email declining a meeting invitation?",
31 + "I'm planning a weekend trip to Montreal. What should I not miss?",
32 + "How do I politely tell my roommate they're too loud at night?",
33 + "What are some tips for staying motivated when learning a new language?",
34 + "My laptop feels slow. What are the first things I should check?",
35 + "Suggest a simple weekly meal-prep plan for someone who works long hours."
36 + ],
37 + "french": [
38 + "Explique la différence entre l'imparfait et le passé composé avec des exemples.",
39 + "Rédige un court paragraphe décrivant l'hiver à Québec.",
40 + "Quels sont les principaux fleuves de France et où se jettent-ils ?",
41 + "Traduis en français : 'The experiment failed, but the data revealed something unexpected.'",
42 + "Écris une recette simple de soupe à l'oignon en cinq étapes.",
43 + "Résume en trois phrases l'intrigue du Petit Prince.",
44 + "Explique ce qu'est la francophonie à un enfant de dix ans.",
45 + "Rédige une lettre de motivation courte pour un stage en informatique."
46 + ],
47 + "english": [
48 + "Write a short paragraph describing a lighthouse at dawn.",
49 + "Summarize the causes of the French Revolution in five sentences.",
50 + "Explain the difference between 'affect' and 'effect' with examples.",
51 + "Describe the water cycle as if teaching a ten-year-old.",
52 + "Write the opening paragraph of a mystery novel set in a small coastal town.",
53 + "Explain what photosynthesis is and why it matters for life on Earth.",
54 + "Draft a short product description for a mechanical keyboard.",
55 + "Describe the plot of Romeo and Juliet in one paragraph."
56 + ],
57 + "reasoning": [
58 + "If all bloops are razzies and all razzies are lazzies, are all bloops necessarily lazzies? Explain step by step.",
59 + "Alice is taller than Bob. Carol is shorter than Bob. Dave is taller than Alice. Order them by height and explain.",
60 + "A farmer needs to cross a river with a wolf, a goat, and a cabbage. The boat holds only one item at a time. How does he do it?",
61 + "Three boxes are labeled 'apples', 'oranges', 'mixed' — all labels are wrong. You may pick one fruit from one box. How do you relabel all boxes correctly?",
62 + "If it takes 5 machines 5 minutes to make 5 widgets, how long does it take 100 machines to make 100 widgets? Explain the common mistake.",
63 + "You have two ropes that each burn in exactly 60 minutes but non-uniformly. How do you measure 45 minutes?",
64 + "A bat and a ball cost $1.10 together. The bat costs $1.00 more than the ball. How much is the ball? Show the algebra.",
65 + "Why can't you conclude that ice cream causes drowning even though sales and drownings are correlated?"
66 + ]
67 + }
68 +}
modified experiments/micro/expG_decision_stability/benchmark.py +205 −11
@@ -1,32 +1,226 @@
1 +#!/usr/bin/env python3
1 2 # =============================================================================
2 3 # Project : localvm-research
3 4 # File : experiments/micro/expG_decision_stability/benchmark.py
4 −# Purpose : Benchmark runner: Decision stability: how many token decisions are stable before full precision is available
5 +# Purpose : Joint (cheap-pass margin × agreement) matrix across bit-widths —
6 +# the Gate-Zero measurement for margin-gated escalation (G02/G23)
5 7 # Author : Simon-Pierre Boucher
6 8 # Contact : contact@spboucher.ai
7 −# Created : 2026-08-11
8 −# Modified : 2026-08-11
9 −# Platform : macOS / Apple Silicon (arm64)
9 +# Created : 2026-08-12
10 +# Modified : 2026-08-12
11 +# Platform : macOS / Apple Silicon (arm64) — MLX / Metal
10 12 # License : All rights reserved (research code)
11 13 # =============================================================================
14 +"""Experiment G — decision stability (charter §9.G).
12 15
13 −"""Benchmark entry point for expG_decision_stability.
16 +Generates greedy continuations with a bf16 reference model, teacher-forces
17 +low-bit quantized variants over the same sequences, and records per-position
18 +margin/agreement/KL. Outputs the joint matrix, AUROC of margin as a
19 +disagreement detector, and escalation curves.
14 20
15 −Must embed the hardware manifest in all result output
16 −(see benchmarks/hardware_manifest.py) and write results to
17 −results/expG_decision_stability/<timestamp>/.
21 +Usage:
22 + .venv/bin/python benchmark.py [--model mlx-community/Qwen3-1.7B-bf16]
23 + [--gen-tokens 128] [--bits 2,3,4,8] [--per-domain 8]
18 24 """
19 25
26 +from __future__ import annotations
27 +
28 +import argparse
29 +import gc
30 +import json
20 31 import sys
32 +import time
33 +from datetime import datetime, timezone
21 34 from pathlib import Path
22 35
23 −sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "benchmarks"))
36 +import mlx.core as mx
37 +import mlx.nn as nn
38 +import numpy as np
39 +from mlx_lm import load
40 +
41 +REPO_ROOT = Path(__file__).resolve().parents[3]
42 +sys.path.insert(0, str(REPO_ROOT / "benchmarks"))
24 43 from hardware_manifest import collect_manifest # noqa: E402
25 44
26 45
46 +def greedy_generate(model, tokenizer, prompt_ids: list[int], n_tokens: int) -> list[int]:
47 + """Greedy generation without sampling helpers — deterministic, no cache reuse
48 + across prompts. Returns generated token ids."""
49 + tokens = list(prompt_ids)
50 + generated = []
51 + from mlx_lm.models.cache import make_prompt_cache
52 +
53 + cache = make_prompt_cache(model)
54 + inp = mx.array(tokens)[None]
55 + for _ in range(n_tokens):
56 + logits = model(inp, cache=cache)
57 + nxt = int(mx.argmax(logits[0, -1]).item())
58 + if nxt == tokenizer.eos_token_id:
59 + break
60 + generated.append(nxt)
61 + inp = mx.array([[nxt]])
62 + return generated
63 +
64 +
65 +def teacher_forced_stats(model, full_ids: list[int], start: int) -> dict:
66 + """Forward the full sequence once; return per-position stats for positions
67 + predicting tokens at indices [start, len(full_ids)) — i.e., logits at
68 + positions start-1 .. len-2."""
69 + logits = model(mx.array(full_ids)[None])[0] # (T, V)
70 + sel = logits[start - 1 : len(full_ids) - 1].astype(mx.float32)
71 + top2 = mx.topk(sel, 2, axis=-1) # values sorted ascending in MLX topk
72 + argmax = mx.argmax(sel, axis=-1)
73 + logprobs = sel - mx.logsumexp(sel, axis=-1, keepdims=True)
74 + mx.eval(top2, argmax, logprobs)
75 + v = np.array(top2)
76 + margin = v[:, 1] - v[:, 0] if v[0, 1] >= v[0, 0] else v[:, 0] - v[:, 1]
77 + return {
78 + "margin": np.abs(margin),
79 + "argmax": np.array(argmax),
80 + # float16 storage: 48 trajectories × (128, ~152k vocab) would be ~4 GB
81 + # in float32; KL is computed in float32 at use time.
82 + "logprobs": np.array(logprobs).astype(np.float16),
83 + }
84 +
85 +
86 +def auroc(scores: np.ndarray, labels: np.ndarray) -> float:
87 + """AUROC of `scores` (higher = predicted positive) for binary labels.
88 + Here: score = -margin (low margin should predict disagreement=1)."""
89 + pos, neg = scores[labels == 1], scores[labels == 0]
90 + if len(pos) == 0 or len(neg) == 0:
91 + return float("nan")
92 + order = np.argsort(np.concatenate([pos, neg]), kind="mergesort")
93 + ranks = np.empty(len(order)); ranks[order] = np.arange(1, len(order) + 1)
94 + # average ranks for ties
95 + allv = np.concatenate([pos, neg])
96 + sorted_v = allv[order]
97 + i = 0
98 + while i < len(sorted_v):
99 + j = i
100 + while j + 1 < len(sorted_v) and sorted_v[j + 1] == sorted_v[i]:
101 + j += 1
102 + if j > i:
103 + ranks[order[i : j + 1]] = ranks[order[i : j + 1]].mean()
104 + i = j + 1
105 + r_pos = ranks[: len(pos)].sum()
106 + return float((r_pos - len(pos) * (len(pos) + 1) / 2) / (len(pos) * len(neg)))
107 +
108 +
109 +def escalation_curve(margins: np.ndarray, agree: np.ndarray, points: int = 200) -> list[dict]:
110 + """For threshold τ over margins: escalate tokens with margin < τ (assume the
111 + escalated decision becomes correct). Report escalated fraction vs residual
112 + disagreement (disagreements with margin ≥ τ)."""
113 + qs = np.quantile(margins, np.linspace(0, 1, points))
114 + out, n = [], len(margins)
115 + for tau in qs:
116 + esc = margins < tau
117 + residual = np.sum((~esc) & (agree == 0)) / n
118 + out.append({"tau": float(tau), "escalated_frac": float(esc.mean()),
119 + "residual_disagree": float(residual)})
120 + return out
121 +
122 +
27 123 def main() -> None:
28 − manifest = collect_manifest()
29 − raise NotImplementedError("experiment not yet implemented")
124 + ap = argparse.ArgumentParser()
125 + ap.add_argument("--model", default="mlx-community/Qwen3-1.7B-bf16")
126 + ap.add_argument("--gen-tokens", type=int, default=128)
127 + ap.add_argument("--bits", default="2,3,4,8")
128 + ap.add_argument("--per-domain", type=int, default=8)
129 + ap.add_argument("--group-size", type=int, default=64)
130 + args = ap.parse_args()
131 + bits_list = [int(b) for b in args.bits.split(",")]
132 +
133 + prompts_file = REPO_ROOT / "benchmarks" / "datasets" / "eval_prompts.json"
134 + domains = json.loads(prompts_file.read_text())["domains"]
135 +
136 + print(f"loading reference {args.model} …", flush=True)
137 + model, tokenizer = load(args.model)
138 +
139 + # -------- pass 1: reference greedy trajectories + reference stats
140 + trajectories = [] # {domain, prompt_ids, full_ids, start}
141 + t0 = time.time()
142 + for domain, plist in domains.items():
143 + for prompt in plist[: args.per_domain]:
144 + msgs = [{"role": "user", "content": prompt}]
145 + ids = tokenizer.apply_chat_template(msgs, add_generation_prompt=True)
146 + gen = greedy_generate(model, tokenizer, ids, args.gen_tokens)
147 + if len(gen) < 8:
148 + continue
149 + trajectories.append({"domain": domain, "full_ids": list(ids) + gen, "start": len(ids)})
150 + print(f" generated {domain}", flush=True)
151 + print(f"reference generation done in {time.time() - t0:.0f}s "
152 + f"({len(trajectories)} trajectories)", flush=True)
153 +
154 + ref_stats = [teacher_forced_stats(model, t["full_ids"], t["start"]) for t in trajectories]
155 +
156 + # -------- pass 2: quantized variants, teacher-forced on the same ids
157 + per_bits: dict[int, dict] = {}
158 + for bits in bits_list:
159 + print(f"quantizing to {bits}-bit (group {args.group_size}) …", flush=True)
160 + del model
161 + gc.collect(); mx.clear_cache()
162 + model, _ = load(args.model)
163 + nn.quantize(model, group_size=args.group_size, bits=bits,
164 + class_predicate=lambda p, m: isinstance(m, nn.Linear)
165 + and m.weight.shape[-1] % args.group_size == 0)
166 + rows = []
167 + for t, ref in zip(trajectories, ref_stats):
168 + qs = teacher_forced_stats(model, t["full_ids"], t["start"])
169 + ref_next = np.array(t["full_ids"][t["start"]:]) # actual (=ref argmax) tokens
170 + agree = (qs["argmax"] == ref_next).astype(np.int8)
171 + # KL(ref||q) per position
172 + ref_lp = ref["logprobs"].astype(np.float32)
173 + kl = np.sum(np.exp(ref_lp) * (ref_lp - qs["logprobs"].astype(np.float32)), axis=-1)
174 + rows.append({"domain": t["domain"], "margin": qs["margin"],
175 + "agree": agree, "kl": kl})
176 + margins = np.concatenate([r["margin"] for r in rows])
177 + agrees = np.concatenate([r["agree"] for r in rows])
178 + kls = np.concatenate([r["kl"] for r in rows])
179 + disagree = 1 - agrees
180 + stats = {
181 + "bits": bits,
182 + "n_positions": int(len(margins)),
183 + "agreement_rate": float(agrees.mean()),
184 + "mean_kl_ref_q": float(np.mean(kls)),
185 + "auroc_margin_predicts_disagreement": auroc(-margins, disagree),
186 + "median_margin_agree": float(np.median(margins[agrees == 1])),
187 + "median_margin_disagree": float(np.median(margins[agrees == 0])) if (agrees == 0).any() else None,
188 + "escalation_curve": escalation_curve(margins, agrees),
189 + "per_domain": {
190 + d: {
191 + "agreement_rate": float(np.concatenate([r["agree"] for r in rows if r["domain"] == d]).mean()),
192 + "auroc": auroc(
193 + -np.concatenate([r["margin"] for r in rows if r["domain"] == d]),
194 + 1 - np.concatenate([r["agree"] for r in rows if r["domain"] == d]),
195 + ),
196 + }
197 + for d in domains
198 + },
199 + }
200 + # operating point: escalation fraction to reach 99% agreement
201 + for pt in stats["escalation_curve"]:
202 + if pt["residual_disagree"] <= 0.01:
203 + stats["escalation_frac_for_99pct"] = pt["escalated_frac"]
204 + break
205 + per_bits[bits] = stats
206 + print(f" {bits}-bit: agree={stats['agreement_rate']:.4f} "
207 + f"AUROC={stats['auroc_margin_predicts_disagreement']:.3f} "
208 + f"esc@99%={stats.get('escalation_frac_for_99pct', 'n/a')}", flush=True)
209 +
210 + ts = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
211 + out_dir = REPO_ROOT / "results" / "expG_decision_stability" / ts
212 + out_dir.mkdir(parents=True)
213 + payload = {
214 + "experiment": "expG_decision_stability",
215 + "author": "Simon-Pierre Boucher",
216 + "contact": "contact@spboucher.ai",
217 + "manifest": collect_manifest(),
218 + "config": vars(args),
219 + "n_trajectories": len(trajectories),
220 + "results_by_bits": {str(k): v for k, v in per_bits.items()},
221 + }
222 + (out_dir / "results.json").write_text(json.dumps(payload, indent=2))
223 + print(f"\nwrote {out_dir / 'results.json'}")
30 224
31 225
32 226 if __name__ == "__main__":
modified experiments/micro/expG_decision_stability/hypothesis.md +42 −8
@@ -3,31 +3,65 @@ project: localvm-research
3 3 document: expG_decision_stability/hypothesis
4 4 author: Simon-Pierre Boucher
5 5 contact: contact@spboucher.ai
6 −created: 2026-08-11
6 +created: 2026-08-12
7 7 status: draft
8 8 ---
9 9
10 10 # Hypothesis — expG_decision_stability
11 11
12 +The Gate-Zero experiment (candidate_ranking.md C4): measures the joint
13 +distribution of (cheap-pass top-1 margin × agreement with the full-precision
14 +model) that the literature does not report, and which decides whether
15 +margin-gated escalation (G02 MarginGate, and with it candidate C1) is viable.
16 +
12 17 ```text
13 18 Hypothesis
14 − <what we believe and why>
19 + For a small pretrained LLM, the top-1 logit margin of a low-bit quantized
20 + variant is a strong predictor of whether its greedy token decision matches
21 + the full-precision model: disagreements concentrate at small margins.
22 + Concretely, margin achieves AUROC ≥ 0.8 as a disagreement detector at
23 + 3–4 bits, so that escalating only the low-margin fraction of tokens
24 + (expected 10–30%) recovers ≥98–99% greedy agreement.
15 25
16 26 Falsification criterion
17 − <the concrete measurable outcome that would prove this wrong>
27 + If AUROC < 0.65 at every bit-width, or if reaching 99% agreement requires
28 + escalating > 50% of tokens at 4-bit (i.e., disagreements are NOT
29 + concentrated at low margins), then decision-uncertainty is not a usable
30 + gating signal: G02/G23 die and candidate C1 must be redesigned around
31 + amortized verification (C2) instead.
18 32
19 33 Method
20 − <exact procedure, model(s), data, seeds, measurement points>
34 + Model: Qwen3-1.7B (bf16 reference; representative modern architecture,
35 + runs fully in memory). Variants: MLX affine quantization at 2, 3, 4, 8
36 + bits (group size 64), quantizing all linear layers.
37 + For each of 48 prompts across 6 domains (code, math, chat, FR, EN,
38 + reasoning; benchmarks/datasets/eval_prompts.json): generate 128 tokens
39 + greedily with the bf16 reference, then teacher-force every variant over
40 + the same token sequence and record, per position: variant top-1 margin
41 + (logit gap between its top-2 candidates), variant argmax, reference
42 + argmax, agreement bit, and KL(ref||variant).
43 + Analysis: per bit-width and per domain — agreement rate; margin
44 + distributions conditioned on agree/disagree; AUROC of margin as a
45 + disagreement detector; escalation curve (fraction of tokens with margin
46 + below τ vs residual disagreement rate after escalating those tokens);
47 + the operating point (escalation fraction) that reaches 99% agreement.
48 + Seed fixed (greedy decode; deterministic). Hardware manifest embedded.
21 49
22 50 Baseline
23 − <what this is compared against — no straw men>
51 + The 8-bit variant (known near-lossless: expected ≥99% agreement without
52 + any gating) anchors the top of the scale; published aggregate agreement
53 + rates for 4-bit (~90–91%, llama.cpp 'same top p' data) sanity-check our
54 + pipeline. No straw men: the comparison is against unconditional
55 + escalation (loading full weights for all tokens).
24 56
25 57 Result
26 − <filled after the run: numbers, with mean/median/std and run count>
58 + <filled after the run>
27 59
28 60 Interpretation
29 − <what the numbers mean; alternative explanations considered>
61 + <filled after the run>
30 62
31 63 Next experiment
32 − <the most informative follow-up given this result>
64 + <filled after the run; if AUROC high → expD (progressive reconstruction:
65 + does escalation *bytes* stay small?); if low → G24 BasinCollapse check
66 + before abandoning>
33 67 ```
34 68