SPB Git

spb/localvm-research Public License

Running LLMs larger than memory on a consumer Mac — falsification-driven research: margin-gated deferred refinement, out-of-core verification on Apple Silicon. TR-01 published.

Python 63.2% JavaScript 23.5% CSS 11.8% Shell 0.9% Makefile 0.5%

Bootstrap: charter-compliant skeleton, header enforcement, hardware manifest, experiment scaffolds

- Repository layout per CLAUDE.md §3
- tools/check_headers.py enforcing §0.1 (58 files pass)
- tools/new_experiment.py scaffolding with seven-field hypothesis block (§10)
- benchmarks/hardware_manifest.py: macOS-native fingerprint (M5 Max / 48GB verified)
- Eight micro-experiment scaffolds (§9, exp A–H)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simon-Pierre Boucher committed 5 h ago (Aug 12, 2026)

Showing 61 changed files with +2,393 and −0

added .gitignore +46 −0
@@ -0,0 +1,46 @@
1 +# =============================================================================
2 +# Project : localvm-research
3 +# File : .gitignore
4 +# Purpose : Git ignore rules (venvs, caches, large artifacts, OS noise)
5 +# Author : Simon-Pierre Boucher
6 +# Contact : contact@spboucher.ai
7 +# Created : 2026-08-11
8 +# Modified : 2026-08-11
9 +# Platform : macOS / Apple Silicon (arm64)
10 +# License : All rights reserved (research code)
11 +# =============================================================================
12 +
13 +# Python
14 +__pycache__/
15 +*.py[cod]
16 +.venv/
17 +venv/
18 +*.egg-info/
19 +.pytest_cache/
20 +.ruff_cache/
21 +.mypy_cache/
22 +
23 +# macOS
24 +.DS_Store
25 +
26 +# Build artifacts
27 +build/
28 +*.metallib
29 +*.air
30 +*.o
31 +*.dylib
32 +
33 +# Large model files — never commit checkpoints
34 +*.safetensors
35 +*.gguf
36 +*.bin
37 +*.pt
38 +*.pth
39 +models_cache/
40 +
41 +# Local editor/session
42 +.claude/settings.local.json
43 +
44 +# Raw traces can be huge; committed results are curated JSON/CSV/plots
45 +results/**/*.trace
46 +results/**/*.fsusage
added CITATION.cff +20 −0
@@ -0,0 +1,20 @@
1 +# =============================================================================
2 +# Project : localvm-research
3 +# File : CITATION.cff
4 +# Purpose : Citation metadata for the research repository
5 +# Author : Simon-Pierre Boucher
6 +# Contact : contact@spboucher.ai
7 +# Created : 2026-08-11
8 +# Modified : 2026-08-11
9 +# Platform : macOS / Apple Silicon (arm64)
10 +# License : All rights reserved (research code)
11 +# =============================================================================
12 +cff-version: 1.2.0
13 +message: "If you use this research, please cite it as below."
14 +title: "localvm-research: out-of-core execution of large pretrained LLMs on consumer Apple Silicon"
15 +authors:
16 + - family-names: "Boucher"
17 + given-names: "Simon-Pierre"
18 + email: "contact@spboucher.ai"
19 +date-released: "2026-08-11"
20 +version: "0.0.1"
added CLAUDE.md +781 −0
@@ -0,0 +1,781 @@
1 +# CLAUDE.md
2 +
3 +**Project codename:** `localvm-research` (provisional — final name decided by evidence, not preference)
4 +**Principal investigator / Author:** Simon-Pierre Boucher — <contact@spboucher.ai>
5 +**Primary platform:** Apple Silicon Mac (macOS 14+)
6 +**Document status:** Living research charter. Claude must treat this as the authoritative project specification.
7 +
8 +---
9 +
10 +## 0. Administrative conventions (MANDATORY — read before anything else)
11 +
12 +### 0.1 Author header requirement
13 +
14 +**Every single source file created in this project — without exception — must begin with a standardized author header.**
15 +
16 +This applies to: Python, C++, Objective-C, Swift, Metal shaders, shell scripts, Makefiles, CMake files, configuration files that support comments, and benchmark scripts.
17 +
18 +#### Python / shell / YAML / TOML header
19 +
20 +```python
21 +# =============================================================================
22 +# Project : localvm-research
23 +# File : <relative/path/to/file.py>
24 +# Purpose : <one-line description of what this file does>
25 +# Author : Simon-Pierre Boucher
26 +# Contact : contact@spboucher.ai
27 +# Created : <YYYY-MM-DD>
28 +# Modified : <YYYY-MM-DD>
29 +# Platform : macOS / Apple Silicon (arm64)
30 +# License : All rights reserved (research code)
31 +# =============================================================================
32 +```
33 +
34 +#### C++ / Metal / Swift / Objective-C header
35 +
36 +```cpp
37 +// ============================================================================
38 +// Project : localvm-research
39 +// File : <relative/path/to/file.cpp>
40 +// Purpose : <one-line description>
41 +// Author : Simon-Pierre Boucher
42 +// Contact : contact@spboucher.ai
43 +// Created : <YYYY-MM-DD>
44 +// Modified : <YYYY-MM-DD>
45 +// Platform : macOS / Apple Silicon (arm64) — Metal / Accelerate / MLX
46 +// License : All rights reserved (research code)
47 +// ============================================================================
48 +```
49 +
50 +#### Markdown research documents header (front matter)
51 +
52 +```markdown
53 +---
54 +project: localvm-research
55 +document: <name>
56 +author: Simon-Pierre Boucher
57 +contact: contact@spboucher.ai
58 +created: <YYYY-MM-DD>
59 +status: draft | reviewed | final
60 +---
61 +```
62 +
63 +Rules:
64 +
65 +1. The header must be the **first content** of the file (after a shebang line if one exists).
66 +2. `Modified` must be updated whenever the file is substantially changed.
67 +3. A CI-style check script (`tools/check_headers.py`) must be written early in the project and run before every commit; it fails if any tracked source file lacks a conforming header.
68 +4. Generated files (e.g., results JSON) are exempt, but the generator must embed `"author": "Simon-Pierre Boucher", "contact": "contact@spboucher.ai"` in output metadata where format permits.
69 +
70 +### 0.2 macOS-first design constraint
71 +
72 +**Everything in this project must be designed to run on a Mac.** Specifically:
73 +
74 +* Target machine class: Apple Silicon (M1/M2/M3/M4 family), 16–64 GB unified memory, internal NVMe SSD.
75 +* Default compute paths: **MLX**, **Metal (MSL kernels)**, **Accelerate/BLAS**, **PyTorch with MPS backend**, plain CPU (arm64 NEON).
76 +* CUDA is allowed **only** as an optional, clearly isolated validation path (`src/backends/cuda_optional/`), never a dependency of the core runtime.
77 +* All I/O experiments must account for macOS specifics: APFS behavior, the unified page cache, `mmap` semantics on macOS, `F_NOCACHE`/`fcntl` instead of Linux `O_DIRECT`, `posix_fadvise` absence, `purge`/`vm_stat` for memory pressure measurement, and Apple SSD controller characteristics.
78 +* Instrumentation must use macOS-native sources where possible: `task_info` / `mach` APIs, `vm_stat`, `fs_usage`, `powermetrics` (energy, requires sudo — degrade gracefully), Instruments traces where practical, `sysctl hw.*` for hardware discovery.
79 +* Build tooling: `uv` or `pip` + `venv` for Python; `cmake` + AppleClang for C++; `xcrun -sdk macosx metal` for Metal shader compilation. No Linux-only assumptions anywhere in the core.
80 +* A hardware manifest (`benchmarks/hardware_manifest.py`) must record chip model, core counts (P/E), GPU core count, memory size, SSD model, macOS version, and software versions into every result file.
81 +
82 +### 0.3 Repository discipline
83 +
84 +* Git from day one. Meaningful commits. No giant "misc" commits.
85 +* Every experiment result must be reproducible from: (commit hash) + (config file) + (seed) + (hardware manifest).
86 +* Python: type hints, `ruff` for lint/format, `pytest` for correctness tests of numerical code.
87 +* C++: `-Wall -Wextra -Werror`, sanitizers in debug builds.
88 +* No result may be reported from an uncommitted working tree.
89 +
90 +---
91 +
92 +## 1. Mission
93 +
94 +Your objective is to investigate, design, implement, and experimentally validate a genuinely new way to run **existing pretrained open-weight large language models that normally do not fit comfortably in consumer-Mac memory** on ordinary local Apple Silicon hardware.
95 +
96 +The target problem is **not**:
97 +
98 +* training a new foundation model;
99 +* designing a smaller replacement model;
100 +* merely applying standard quantization;
101 +* merely using CPU offload;
102 +* merely using SSD swap;
103 +* merely pruning a model once and accepting permanent quality loss;
104 +* or reproducing llama.cpp, MLX, PowerInfer, FlexGen, speculative decoding, or another existing system.
105 +
106 +The target is broader and more ambitious:
107 +
108 +> **Given an already-trained model whose normal memory and bandwidth requirements exceed the target Mac, determine whether its execution can be reorganized, transformed, compiled, approximated, decomposed, scheduled, cached, paged, predicted, refined, or otherwise executed differently so that the model remains meaningfully useful on consumer Apple Silicon hardware with dramatically lower resident memory and/or memory bandwidth requirements.**
109 +
110 +Do not assume in advance what the solution should be.
111 +Do not force a particular architecture.
112 +Do not begin implementation before understanding the state of the art.
113 +Your job is to discover what is actually possible.
114 +
115 +---
116 +
117 +## 2. Core research question
118 +
119 +Investigate:
120 +
121 +> **Can an existing dense or MoE pretrained LLM be transformed post-training into an execution representation whose instantaneous working set is dramatically smaller than the full checkpoint, while preserving most or all of the original model's useful capabilities?**
122 +
123 +The most important distinction is:
124 +
125 +```text
126 +total model size
127 +≠
128 +resident model size
129 +≠
130 +bytes read per token
131 +≠
132 +parameters materially required for a particular token
133 +```
134 +
135 +Explore whether these quantities can be decoupled.
136 +
137 +A successful system could potentially make a model much larger than available unified memory usable locally on a Mac, where additional model size primarily increases latency or storage requirements rather than making execution impossible.
138 +
139 +### Guiding principle
140 +
141 +Do not optimize first for benchmark scores. Initially optimize for discovering whether there is a **new execution regime**.
142 +
143 +The main scientific question is:
144 +
145 +```text
146 +How little of the original model must actually be represented,
147 +loaded, reconstructed, or evaluated to reproduce its behavior?
148 +```
149 +
150 +Treat this as a systems + machine-learning research problem. You are free to discover that the initial premise is wrong. **Negative experimental results are useful.**
151 +
152 +---
153 +
154 +## 3. Project structure (real research-project layout)
155 +
156 +The repository must follow this layout. Create the skeleton (with header-compliant placeholder files) before Phase 1 concludes.
157 +
158 +```text
159 +localvm-research/
160 +├── CLAUDE.md # this charter
161 +├── README.md # public-facing summary (written last, updated continuously)
162 +├── CITATION.cff # citation metadata (author: Simon-Pierre Boucher)
163 +├── LICENSE
164 +├── pyproject.toml # Python project config (ruff, pytest, deps)
165 +├── Makefile # top-level tasks: setup, lint, test, bench, headers
166 +│
167 +├── research/ # the scientific paper trail
168 +│ ├── LOG.md # dated research log (append-only)
169 +│ ├── state_of_the_art.md # Phase 2 deliverable
170 +│ ├── research_gaps.md # Phase 3 deliverable
171 +│ ├── candidate_ranking.md # Phase 4 deliverable
172 +│ ├── bibliography.md # every source, with links and access dates
173 +│ ├── novelty_check.md # Phase 11 deliverable
174 +│ └── notes/ # per-topic reading notes (one file per theme)
175 +│
176 +├── src/ # core library code
177 +│ ├── localvm/
178 +│ │ ├── __init__.py
179 +│ │ ├── models/ # model loading, checkpoint parsing (safetensors/GGUF)
180 +│ │ ├── transforms/ # quantization, decomposition, encodings
181 +│ │ ├── runtime/ # paging, scheduling, caching, prediction
182 +│ │ ├── backends/
183 +│ │ │ ├── mlx_backend/ # MLX / Metal primary path
184 +│ │ │ ├── mps_backend/ # PyTorch-MPS path
185 +│ │ │ ├── cpu_backend/ # Accelerate / NEON path
186 +│ │ │ └── cuda_optional/ # isolated, optional
187 +│ │ ├── instrumentation/ # macOS-native measurement (mach, vm_stat, fs_usage…)
188 +│ │ └── quality/ # perplexity, KL, decision-stability metrics
189 +│ └── kernels/ # custom Metal (.metal) and C++ kernels
190 +│
191 +├── experiments/ # micro-experiments and candidate prototypes
192 +│ ├── micro/
193 +│ │ ├── expA_weight_concentration/
194 +│ │ ├── expB_token_stability/
195 +│ │ ├── expC_semantic_locality/
196 +│ │ ├── expD_progressive_reconstruction/
197 +│ │ ├── expE_partial_gemm/
198 +│ │ ├── expF_error_accumulation/
199 +│ │ ├── expG_decision_stability/
200 +│ │ └── expH_ssd_feasibility/
201 +│ ├── candidate_01/
202 +│ ├── candidate_02/
203 +│ └── candidate_03/
204 +│ └── (each candidate contains:)
205 +│ ├── README.md
206 +│ ├── hypothesis.md
207 +│ ├── implementation/
208 +│ ├── benchmark.py
209 +│ ├── results/
210 +│ └── analysis.md
211 +│
212 +├── benchmarks/ # baselines and standardized harness
213 +│ ├── harness.py # unified benchmark runner
214 +│ ├── hardware_manifest.py # macOS hardware/software fingerprinting
215 +│ ├── baselines/ # llama.cpp, MLX, mmap, offload configs
216 +│ └── datasets/ # eval prompts: code, math, chat, FR, EN, reasoning
217 +│
218 +├── results/ # raw + aggregated results (JSON/CSV + plots)
219 +│ └── <experiment_id>/<timestamp>/
220 +│
221 +├── tools/
222 +│ ├── check_headers.py # enforces §0.1
223 +│ ├── new_experiment.py # scaffolds a compliant experiment directory
224 +│ └── report.py # generates result tables/plots
225 +│
226 +└── docs/ # architecture docs, diagrams, final writeups
227 +```
228 +
229 +Every experiment directory is scaffolded by `tools/new_experiment.py`, which auto-inserts the author header and a `hypothesis.md` template containing the seven-field scientific-discipline block (§10).
230 +
231 +---
232 +
233 +## 4. Phase 1 — Ultra-deep web and literature research
234 +
235 +Before proposing an architecture, perform an extremely deep search of current literature, repositories, technical reports, conference papers, preprints, blog posts, implementations, issue discussions, benchmarks, and systems research.
236 +
237 +The search must be current. Search broadly and recursively. Do not stop after finding several obvious papers. Follow citations backward and forward whenever useful.
238 +
239 +Search arXiv, OpenReview, conference proceedings (MLSys, OSDI, SOSP, ASPLOS, ISCA, NeurIPS, ICML, ICLR, ACL), GitHub, Hugging Face, systems research venues, vendor engineering documentation (especially **Apple ML/Metal engineering material and MLX repos**), academic project pages, and relevant technical discussions.
240 +
241 +Prefer primary sources. For every important technique, find the actual paper and, whenever available, the implementation. Log every consulted source in `research/bibliography.md` with URL and access date.
242 +
243 +### Areas that must be investigated
244 +
245 +At minimum investigate all of the following, while remaining open to unrelated approaches.
246 +
247 +#### 4.1 Quantization
248 +
249 +PTQ; QAT; 8-bit; 6-bit; 4-bit; 3-bit; 2-bit; 1.58-bit; ternary weights; 1-bit approaches; mixed precision; per-layer precision; per-channel precision; per-token precision; dynamic precision; progressive precision; residual quantization; recurrent residual quantization; additive quantization; vector quantization; lattice quantization; codebook methods; weight-only quantization; activation quantization; KV-cache quantization; extreme low-bit inference.
250 +
251 +Study quality degradation and actual memory-bandwidth effects **separately**. Pay particular attention to which formats have efficient Metal/MLX kernels versus which exist only on CUDA.
252 +
253 +#### 4.2 Activation sparsity
254 +
255 +Contextual sparsity; dynamic activation sparsity; FFN sparsity; neuron activation prediction; top-k neuron selection; ReLUfication; SwiGLU sparsification; structured sparsity; unstructured sparsity; N:M sparsity; activation predictors; learned sparsity routers; post-training induced sparsity.
256 +
257 +Study systems including but not limited to: DejaVu; ShadowLLM; PowerInfer; PowerInfer-2; DynamicInfer; SparQ; contextual sparsity systems. Determine exactly what is already known, and what portions assume discrete-GPU architectures that do not map to unified memory.
258 +
259 +#### 4.3 Weight sparsity and pruning
260 +
261 +SparseGPT; Wanda; magnitude pruning; structured pruning; block pruning; channel pruning; layer dropping; width pruning; depth pruning; one-shot pruning; dynamic pruning; recoverable pruning; reversible pruning; conditional pruning.
262 +
263 +Ask whether discarded weights could instead live on the Mac's NVMe SSD and be recovered only when needed.
264 +
265 +#### 4.4 Out-of-core inference
266 +
267 +CPU/GPU offload; RAM/VRAM tiering; NVMe offload; mmap; asynchronous I/O; direct I/O; page cache behavior; pinned memory; unified memory; **Apple Silicon memory behavior specifically**; GPU prefetch; tensor paging; weight streaming; SSD-to-GPU pipelines; near-storage inference; computational storage.
268 +
269 +Investigate systems such as: FlexGen; DeepSpeed inference; ZeRO-Inference; llama.cpp; MLX; PowerInfer; M2Cache; SolidAttention; Petals where relevant; distributed consumer inference systems.
270 +
271 +Measure whether the true bottleneck is capacity, bandwidth, latency, random I/O, or synchronization — **on macOS/APFS/Apple NVMe, not on Linux assumptions**.
272 +
273 +#### 4.5 Model decomposition
274 +
275 +Research whether pretrained weights can be represented as `base + residual`, `shared component + layer-specific correction`, or `low-rank component + sparse residual`.
276 +
277 +Investigate: SVD; truncated SVD; tensor decomposition; low-rank factorization; LoRA-like decomposition of existing weights; cross-layer sharing; DeltaLLM; matrix dictionaries; learned codebooks; basis decomposition; Kronecker decomposition; tensor trains; product quantization; weight clustering; block-level factorization; shared latent weight representations.
278 +
279 +Determine whether model layers contain exploitable redundancy that existing runtimes ignore.
280 +
281 +#### 4.6 Progressive and approximate computation
282 +
283 +Progressive inference; anytime neural networks; early exit; adaptive computation; dynamic depth; conditional computation; residual refinement; coarse-to-fine inference; iterative refinement; approximate matrix multiplication; error-bounded GEMM; approximate computing; lazy tensor evaluation; partial matrix multiplication; adaptive precision numerical computing.
284 +
285 +Do not limit this search to LLM research. Look at computer architecture, numerical linear algebra, databases, graphics, signal processing, scientific computing, and HPC. Potentially useful concepts may already exist outside machine learning.
286 +
287 +#### 4.7 Speculative execution
288 +
289 +Speculative decoding; self-speculative decoding; draft models; Medusa-like approaches; verification methods; exact distribution preservation; optimistic execution; rollback; branch prediction; confidence-based speculative computation.
290 +
291 +Ask whether speculation can happen **inside a transformer forward pass**, not only across future tokens.
292 +
293 +#### 4.8 Memory systems (outside AI literature)
294 +
295 +Virtual memory; demand paging; working-set theory; page replacement; TLBs; cache associativity; hardware prefetchers; branch predictors; speculative execution; NUMA; memory compression (including **macOS compressed memory**); compressed caches; tiered memory; CXL memory; object stores; database buffer pools; columnar execution; query optimizers.
296 +
297 +Ask: *What ideas from operating systems and CPU architecture have never been properly translated to neural-weight execution?*
298 +
299 +#### 4.9 Numerical error analysis
300 +
301 +Interval arithmetic; affine arithmetic; probabilistic bounds; matrix norm bounds; perturbation theory; Lipschitz bounds; error propagation; certified neural networks; robustness certification; approximate linear algebra; bounds for quantized networks.
302 +
303 +A potentially important direction: can computation terminate when additional accuracy can no longer meaningfully affect the output? Do not assume this is feasible. Test it.
304 +
305 +#### 4.10 Output-decision stability
306 +
307 +Investigate the distinction between *reproducing exact hidden states* and *producing the same useful output*.
308 +
309 +For greedy decoding, investigate whether a token decision can be certified without fully reconstructing every upstream operation. For sampling, investigate whether approximate intermediate computation can still preserve or closely approximate the original probability distribution.
310 +
311 +Explore: top-1 logit margins; top-k stability; KL divergence; total variation distance; distributional guarantees; acceptance/rejection correction; speculative verification.
312 +
313 +This is an important area but is **not prescribed as the final approach**.
314 +
315 +---
316 +
317 +## 5. Phase 2 — State-of-the-art map
318 +
319 +Produce `research/state_of_the_art.md` (with the §0.1 front-matter header).
320 +
321 +It must organize every relevant technique by:
322 +
323 +```text
324 +Technique
325 +Problem addressed
326 +Model modification required?
327 +Retraining required?
328 +Memory reduction
329 +Bandwidth reduction
330 +Compute reduction
331 +Latency effect
332 +Quality degradation
333 +Hardware assumptions (explicitly: does it work on Apple Silicon / Metal?)
334 +Open-source implementation (and whether it builds on macOS arm64)
335 +Main limitation
336 +Opportunity for extension
337 +```
338 +
339 +Do not simply summarize papers. Identify where approaches overlap. Identify combinations that have already been tried. Identify ideas that initially seem novel but are actually already known.
340 +
341 +---
342 +
343 +## 6. Phase 3 — Identify genuine gaps
344 +
345 +Produce `research/research_gaps.md`.
346 +
347 +For every promising gap explain:
348 +
349 +1. what existing systems do;
350 +2. what they do not do;
351 +3. why the missing capability might matter;
352 +4. whether there is a plausible mathematical or systems reason it could work;
353 +5. the biggest reason it might fail;
354 +6. the smallest experiment capable of falsifying it — **runnable on a Mac**.
355 +
356 +Generate many candidate ideas. Aim for at least **20 substantially different approaches**. Do not make them superficial variants of the same idea.
357 +
358 +Example categories (non-binding): execution reordering; learned weight paging; progressive reconstruction; activation-conditioned decompression; semantic caches; low-rank hot path + residual cold path; predictive SSD reads; dynamic precision; partial GEMM; temporary model specialization; token-dependent model materialization; model-state compression; weight synthesis; cross-layer reuse; hidden-state approximation; error-controlled execution; reversible approximation; adaptive layer reconstruction.
359 +
360 +These are examples only. Do not constrain the search to them.
361 +
362 +---
363 +
364 +## 7. Phase 4 — Rank candidate approaches
365 +
366 +Create `research/candidate_ranking.md`.
367 +
368 +Score every idea 1–10, with written reasoning, on:
369 +
370 +```text
371 +Novelty
372 +Technical plausibility
373 +Expected memory reduction
374 +Expected bandwidth reduction
375 +Expected quality retention
376 +Implementation complexity
377 +Hardware practicality on Apple Silicon (Metal/MLX kernel feasibility, unified memory fit)
378 +Post-training compatibility
379 +Potential research significance
380 +Risk
381 +```
382 +
383 +Select roughly **3–5 strongest candidates** for experiments.
384 +
385 +Prefer ideas that could fundamentally change the scaling relationship between `model size` and `required resident memory / bytes transferred per token`.
386 +
387 +---
388 +
389 +## 8. Phase 5 — Experimental framework
390 +
391 +Build an experimental framework **before** attempting large models. The system must make experiments reproducible.
392 +
393 +Preferred languages and stacks (Mac-first):
394 +
395 +```text
396 +Python (MLX, PyTorch-MPS, NumPy) for research tooling
397 +C++ (AppleClang, Accelerate) where low-level control matters
398 +Metal Shading Language for custom Apple GPU kernels
399 +Swift/Objective-C shims only where mach/IOKit APIs require them
400 +CUDA only as isolated optional validation on non-Mac hardware
401 +```
402 +
403 +Do not prematurely optimize. Correctness and measurement come first.
404 +
405 +### 8.1 Initial model sizes
406 +
407 +Start small. Use models in approximately this progression:
408 +
409 +```text
410 +0.5B–1B → 3B → 7B–8B → 14B → 32B
411 +```
412 +
413 +Only move to 70B+ if evidence supports it. Prefer modern open-weight architectures representative of models we ultimately want to run, with checkpoints available in safetensors and GGUF.
414 +
415 +### 8.2 Instrumentation (macOS-native)
416 +
417 +The framework must measure, when applicable:
418 +
419 +```text
420 +resident RAM (mach task_info) peak RAM
421 +mapped memory GPU/unified memory in use
422 +SSD reads (fs_usage / iostat) SSD bytes/token
423 +RAM bytes/token effective weight bytes/token
424 +tokens/second time-to-first-token
425 +per-token latency CPU utilization (P vs E cores)
426 +GPU utilization SSD utilization
427 +energy via powermetrics (if sudo) cache hit rate
428 +page faults (vm_stat deltas) number of parameters touched
429 +number of blocks touched effective precision used
430 +recomputation count prediction hit rate
431 +memory-pressure / compressor activity thermal state (throttling detection)
432 +```
433 +
434 +Also measure quality:
435 +
436 +```text
437 +perplexity logit correlation
438 +KL divergence same greedy token rate
439 +top-k overlap task benchmark accuracy
440 +generation similarity long-context behavior
441 +coding behavior reasoning behavior
442 +```
443 +
444 +### 8.3 Most important systems metric
445 +
446 +Track **BYTES READ PER GENERATED TOKEN** separately from model size.
447 +Also track **ACTIVE / TOUCHED PARAMETERS PER TOKEN** where meaningful.
448 +
449 +A system that stores a 150 GB checkpoint but reads only 5 GB per generated token may be much more interesting than one that compresses the checkpoint to 70 GB but reads all 70 GB every token.
450 +
451 +### 8.4 Baselines
452 +
453 +At minimum compare against appropriate configurations of:
454 +
455 +```text
456 +full precision where practical
457 +Q8 / Q6 / Q5 / Q4 / Q3 / Q2 (where practical)
458 +llama.cpp (Metal build)
459 +MLX (native)
460 +CPU offloading
461 +mmap streaming
462 +standard SSD offloading
463 +```
464 +
465 +Also reproduce relevant published methods when feasible. Do not claim improvement against straw-man baselines.
466 +
467 +---
468 +
469 +## 9. Phase 6 — Micro-experiments
470 +
471 +Before implementing a large runtime, answer fundamental questions empirically. Each lives in `experiments/micro/` with the standard scaffold.
472 +
473 +### Experiment A — Weight contribution concentration
474 +For each transformer layer and token, measure how much of the output norm arises from subsets of weight blocks. Ask: can 10%, 20%, 40%, or 60% of blocks reproduce most of the layer output? Test across: code, math, chat, French, English, reasoning, random text, long context.
475 +
476 +### Experiment B — Stability across consecutive tokens
477 +Measure overlap between important blocks at tokens t, t+1, t+2. Compute Jaccard similarity, transition probabilities, working-set lifetime. Determine whether weight access is predictable.
478 +
479 +### Experiment C — Cross-prompt semantic locality
480 +Ask whether prompts belonging to the same semantic domain repeatedly rely on similar regions. If so, investigate whether domain-specific weight caches are possible.
481 +
482 +### Experiment D — Progressive weight reconstruction
483 +Represent weights using multiple progressive approximations (2-bit only; 2+residual; 2+2 residuals; …). For each token/layer determine how rapidly hidden-state error, logit error, and the token decision converge.
484 +
485 +### Experiment E — Partial matrix multiplication
486 +Compute only selected weight blocks. Measure quality vs blocks evaluated. Then determine whether important blocks can be predicted cheaply. Implement at least one Metal kernel variant to check that block-skipping actually saves bandwidth on Apple GPUs (not just FLOPs on paper).
487 +
488 +### Experiment F — Error accumulation
489 +Introduce controlled approximation at individual layers. Determine which layers tolerate error, which amplify error, which recover from error. Map model sensitivity.
490 +
491 +### Experiment G — Decision stability
492 +For an approximate forward pass, compare top logits against the exact model. Ask whether many token decisions become stable before full model precision is available.
493 +
494 +### Experiment H — SSD feasibility (macOS-specific)
495 +Simulate realistic storage on the actual Mac's internal NVMe. Do NOT rely on theoretical bandwidth. Measure actual random and sequential reads at 4 KB, 16 KB, 64 KB, 256 KB, 1 MB, 4 MB — with and without page cache (`F_NOCACHE`), cold vs warm APFS state. Measure concurrent reads while Metal GPU compute occurs. Test whether useful prefetch overlap exists. Record SSD model and thermal state; Apple SSDs throttle.
496 +
497 +---
498 +
499 +## 10. Scientific discipline
500 +
501 +For every experiment explicitly write, in its `hypothesis.md` / `analysis.md`:
502 +
503 +```text
504 +Hypothesis
505 +Falsification criterion
506 +Method
507 +Baseline
508 +Result
509 +Interpretation
510 +Next experiment
511 +```
512 +
513 +Avoid confirmation bias. If an idea fails, record why. **Do not silently discard failures.**
514 +
515 +### Evidence standard
516 +
517 +Never write "this is faster / novel / better / should scale" without evidence. Use benchmark numbers. Whenever possible report:
518 +
519 +```text
520 +mean, median, standard deviation, number of runs,
521 +hardware (chip, RAM, SSD, macOS version), model, context length,
522 +generation length, quantization, software versions (MLX, PyTorch, Metal),
523 +thermal conditions
524 +```
525 +
526 +---
527 +
528 +## 11. Phase 7 — Prototype candidate architectures
529 +
530 +For each of the strongest ideas create its own directory:
531 +
532 +```text
533 +experiments/candidate_01/
534 +experiments/candidate_02/
535 +experiments/candidate_03/
536 +```
537 +
538 +Each candidate must include:
539 +
540 +```text
541 +README.md hypothesis.md
542 +implementation/ benchmark.py
543 +results/ analysis.md
544 +```
545 +
546 +All implementation files carry the §0.1 header and must run on the primary Mac target without CUDA.
547 +
548 +---
549 +
550 +## 12. Phase 8 — Automatic research loop
551 +
552 +Operate as a research agent. After every experiment:
553 +
554 +1. analyze results;
555 +2. determine the most informative next experiment;
556 +3. search the literature again if results reveal something unexpected;
557 +4. update hypotheses;
558 +5. modify the candidate architecture;
559 +6. rerun;
560 +7. compare;
561 +8. continue until evidence strongly favors or rejects the approach.
562 +
563 +Do not follow a rigid predefined implementation roadmap if experiments contradict it.
564 +
565 +### Research log
566 +
567 +Maintain `research/LOG.md` (append-only). Every substantial action must contain:
568 +
569 +```text
570 +date/time (local, with timezone)
571 +question
572 +experiment
573 +result
574 +interpretation
575 +decision
576 +```
577 +
578 +This must make the entire reasoning process auditable.
579 +
580 +---
581 +
582 +## 13. Phase 9 — Try to discover a new execution abstraction
583 +
584 +Do not merely seek an optimization. Seek an abstraction that could change how large local models are executed. Useful questions include:
585 +
586 +```text
587 +Could model weights behave like virtual memory?
588 +Could only a token-specific working set be materialized?
589 +Could a model be represented as a cheap approximate core plus recoverable information?
590 +Could computation proceed before all weights are available?
591 +Could missing computation be added later only when needed?
592 +Could weights be reconstructed from shared bases?
593 +Could hidden-state trajectories predict future memory accesses?
594 +Could the system learn its own cache policy?
595 +Could model execution become demand-driven?
596 +Could the runtime optimize for decision stability rather than numerical exactness?
597 +Could a model larger than RAM become usable because RAM limits working-set size
598 +rather than total model size?
599 +Could Apple's unified memory + fast NVMe make an abstraction viable that
600 +discrete-GPU architectures cannot support?
601 +```
602 +
603 +These are research questions, not required design decisions.
604 +
605 +---
606 +
607 +## 14. Phase 10 — Build the best prototype supported by evidence
608 +
609 +Once experiments clearly favor an architecture, build a prototype runtime. A possible CLI shape could eventually be:
610 +
611 +```bash
612 +localvm compile MODEL_PATH \
613 + --target-memory 16GB \
614 + --storage /Volumes/FastNVMe/localvm
615 +```
616 +
617 +then:
618 +
619 +```bash
620 +localvm run COMPILED_MODEL
621 +```
622 +
623 +Naming and architecture should only be finalized after research.
624 +
625 +### Compilation stage
626 +
627 +If useful, compilation may perform offline operations such as: profiling; weight analysis; quantization; factorization; clustering; reordering; block creation; index construction; activation tracing; cache-profile creation; predictor training; error-bound estimation; progressive encoding; layout optimization (aligned to APFS/Metal buffer constraints).
628 +
629 +Compilation can take substantial time. Inference must be the optimized stage.
630 +
631 +### Hard constraint
632 +
633 +The original pretrained model must remain the source model. You may transform its representation post-training. Small auxiliary predictors, indexes, adapters, or calibration passes are allowed if justified. **Do not solve the problem by simply training a new smaller LLM to replace it.**
634 +
635 +---
636 +
637 +## 15. Target hardware
638 +
639 +Primary research target:
640 +
641 +```text
642 +Apple Silicon Mac (laptop or desktop)
643 +16–64 GB unified memory
644 +internal Apple NVMe SSD (2–8 GB/s class)
645 +Metal-capable GPU sharing memory with CPU
646 +```
647 +
648 +Apple Silicon is the primary platform because:
649 +
650 +```text
651 +CPU and GPU share unified memory
652 +Metal allows custom kernels
653 +fast internal SSDs are standard
654 +large memory configurations exist (up to 128–192 GB on desktop parts)
655 +memory compression is built into the OS
656 +```
657 +
658 +The architecture should nevertheless remain conceptually hardware-independent; document (do not implement, unless trivial) how each mechanism would map to x86 + discrete GPU.
659 +
660 +### Stretch target
661 +
662 +```text
663 +model total representation: 100–250+ GB
664 +machine unified memory: 16–32 GB
665 +useful generation speed: interactive or near-interactive
666 +quality: close to original model
667 +```
668 +
669 +Do not assume this target is achievable. The research must establish the actual limits.
670 +
671 +---
672 +
673 +## 16. What counts as a breakthrough
674 +
675 +A result is scientifically interesting if at least one of the following is demonstrated:
676 +
677 +**A.** A model significantly larger than unified memory runs locally on a Mac with acceptable interactive latency.
678 +**B.** Bytes transferred per token become substantially smaller than the compressed checkpoint size.
679 +**C.** Only a small token-dependent fraction of model information is required during typical inference.
680 +**D.** A progressive or conditional execution mechanism preserves model quality while avoiding large amounts of weight loading.
681 +**E.** A new post-training representation creates a qualitatively better storage/RAM/quality tradeoff than fixed quantization.
682 +**F.** A new caching, prediction, scheduling, decomposition, or reconstruction mechanism materially changes out-of-core inference economics.
683 +
684 +## 17. Failure criteria
685 +
686 +Be willing to conclude that an approach does not work. For example:
687 +
688 +```text
689 +SSD bandwidth fundamentally dominates
690 +weight accesses are insufficiently predictable
691 +required working set is nearly the entire model
692 +approximation errors compound uncontrollably
693 +low-bit base models destroy routing information
694 +prediction overhead exceeds savings
695 +random I/O eliminates theoretical advantages
696 +quality degradation is unacceptable
697 +macOS I/O or Metal dispatch overhead erases theoretical wins
698 +```
699 +
700 +If these occur, document them in `research/LOG.md` and the relevant `analysis.md`, then pivot.
701 +
702 +---
703 +
704 +## 18. Deliverables
705 +
706 +Eventually produce (all with compliant headers):
707 +
708 +```text
709 +README.md
710 +CITATION.cff
711 +research/
712 + state_of_the_art.md
713 + research_gaps.md
714 + candidate_ranking.md
715 + novelty_check.md
716 + LOG.md
717 + bibliography.md
718 +src/
719 +experiments/
720 +benchmarks/
721 +results/
722 +docs/
723 +tools/
724 +```
725 +
726 +The final README must explain:
727 +
728 +```text
729 +the problem
730 +what existing systems do
731 +what gap was discovered
732 +the proposed architecture
733 +why it should work
734 +experimental evidence (with numbers, on named Mac hardware)
735 +performance
736 +limitations
737 +how to reproduce (exact Mac setup instructions)
738 +future research
739 +```
740 +
741 +---
742 +
743 +## 19. Phase 11 — Novelty verification
744 +
745 +Before claiming novelty, perform a dedicated final literature search using terminology derived from the architecture actually discovered. Search for conceptual equivalents, not merely identical terminology.
746 +
747 +For example, if the architecture resembles `lazy neural execution`, `progressive weight materialization`, `conditional tensor paging`, or `activation-conditioned decompression` — search every plausible synonym.
748 +
749 +Assume an idea is **not** novel until evidence suggests otherwise. Record the process and conclusion in `research/novelty_check.md`.
750 +
751 +---
752 +
753 +## 20. Most important instruction
754 +
755 +Do not become attached to any particular solution suggested in this document.
756 +
757 +The purpose of this project is **not to implement a preconceived WeightVM, NeuralOS, progressive quantizer, sparse runtime, or paging engine**. Those ideas are merely clues.
758 +
759 +The actual assignment is:
760 +
761 +> **Search deeply enough, reason independently enough, and experiment aggressively enough to discover the best technically plausible way of making an existing large pretrained model usable on a Mac that normally should not be able to run it.**
762 +
763 +Start with literature. Then generate hypotheses. Then falsify them experimentally. Let evidence determine the architecture.
764 +
765 +The ideal outcome is not simply another optimization. The ideal outcome is a mechanism that changes the relationship:
766 +
767 +```text
768 +model size → hardware required
769 +```
770 +
771 +into something closer to:
772 +
773 +```text
774 +instantaneous useful working set → hardware required
775 +```
776 +
777 +while keeping the intelligence already present in the original pretrained model.
778 +
779 +---
780 +
781 +*Author: Simon-Pierre Boucher — contact@spboucher.ai — All research artifacts in this repository carry this attribution.*
added LICENSE +7 −0
@@ -0,0 +1,7 @@
1 +Copyright (c) 2026 Simon-Pierre Boucher <contact@spboucher.ai>
2 +
3 +All rights reserved (research code).
4 +
5 +This repository contains research code and documents. No permission is granted
6 +to use, copy, modify, or distribute this work, in whole or in part, without
7 +prior written authorization from the author.
added Makefile +35 −0
@@ -0,0 +1,35 @@
1 +# =============================================================================
2 +# Project : localvm-research
3 +# File : Makefile
4 +# Purpose : Top-level tasks: setup, lint, test, bench, headers
5 +# Author : Simon-Pierre Boucher
6 +# Contact : contact@spboucher.ai
7 +# Created : 2026-08-11
8 +# Modified : 2026-08-11
9 +# Platform : macOS / Apple Silicon (arm64)
10 +# License : All rights reserved (research code)
11 +# =============================================================================
12 +
13 +PYTHON ?= python3
14 +
15 +.PHONY: setup lint test bench headers manifest
16 +
17 +setup:
18 + $(PYTHON) -m venv .venv
19 + .venv/bin/pip install -e ".[dev,ml]"
20 +
21 +lint:
22 + .venv/bin/ruff check src tools benchmarks experiments
23 + .venv/bin/ruff format --check src tools benchmarks experiments
24 +
25 +test:
26 + .venv/bin/pytest -q
27 +
28 +headers:
29 + $(PYTHON) tools/check_headers.py
30 +
31 +manifest:
32 + $(PYTHON) benchmarks/hardware_manifest.py
33 +
34 +bench:
35 + @echo "benchmarks/harness.py not yet implemented (Phase 5)"; exit 1
added README.md +44 −0
@@ -0,0 +1,44 @@
1 +---
2 +project: localvm-research
3 +document: README
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# localvm-research
11 +
12 +**Can an existing pretrained LLM that does not fit in a consumer Mac's memory be
13 +transformed post-training into an execution representation whose instantaneous
14 +working set is dramatically smaller than the full checkpoint — while preserving
15 +most of its useful capabilities?**
16 +
17 +This repository is a living research project (charter: [CLAUDE.md](CLAUDE.md)).
18 +The central distinction under investigation:
19 +
20 +```text
21 +total model size ≠ resident model size ≠ bytes read per token ≠ parameters required for this token
22 +```
23 +
24 +- **Author:** Simon-Pierre Boucher — <contact@spboucher.ai>
25 +- **Primary platform:** Apple Silicon Mac (unified memory, Metal, internal NVMe), macOS 14+
26 +- **Status:** Phase 1 (literature research) in progress. See `research/LOG.md` for the audit trail.
27 +
28 +## Layout
29 +
30 +| Path | Contents |
31 +|---|---|
32 +| `research/` | Scientific paper trail: log, state of the art, gaps, rankings, bibliography |
33 +| `src/localvm/` | Core library (models, transforms, runtime, backends, instrumentation, quality) |
34 +| `experiments/` | Micro-experiments (A–H) and candidate prototypes |
35 +| `benchmarks/` | Unified harness, hardware manifest, baselines, eval datasets |
36 +| `results/` | Raw + aggregated results (reproducible from commit + config + seed + manifest) |
37 +| `tools/` | `check_headers.py`, `new_experiment.py`, `report.py` |
38 +
39 +## Reproducibility
40 +
41 +Every result is reproducible from: commit hash + config + seed + hardware manifest
42 +(`benchmarks/hardware_manifest.py`). No result is reported from an uncommitted tree.
43 +
44 +*This README is updated continuously and finalized last (charter §18).*
added benchmarks/baselines/README.md +12 −0
@@ -0,0 +1,12 @@
1 +---
2 +project: localvm-research
3 +document: benchmarks/baselines/README
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# benchmarks/baselines
11 +
12 +Baseline configurations: llama.cpp (Metal), MLX native, mmap streaming, offload.
added benchmarks/datasets/README.md +12 −0
@@ -0,0 +1,12 @@
1 +---
2 +project: localvm-research
3 +document: benchmarks/datasets/README
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# benchmarks/datasets
11 +
12 +Eval prompt sets: code, math, chat, FR, EN, reasoning, long context.
added benchmarks/hardware_manifest.py +146 −0
@@ -0,0 +1,146 @@
1 +#!/usr/bin/env python3
2 +# =============================================================================
3 +# Project : localvm-research
4 +# File : benchmarks/hardware_manifest.py
5 +# Purpose : macOS hardware/software fingerprint embedded in every result file
6 +# Author : Simon-Pierre Boucher
7 +# Contact : contact@spboucher.ai
8 +# Created : 2026-08-11
9 +# Modified : 2026-08-11
10 +# Platform : macOS / Apple Silicon (arm64)
11 +# License : All rights reserved (research code)
12 +# =============================================================================
13 +"""Collect a reproducibility manifest for the current Mac.
14 +
15 +Records chip model, P/E core counts, GPU core count, unified memory size,
16 +SSD model, macOS version, and versions of the key software stack (Python,
17 +MLX, PyTorch, NumPy), plus the current git commit. Every benchmark result
18 +JSON must embed this manifest (CLAUDE.md §0.2, §10).
19 +
20 +Usage:
21 + python3 benchmarks/hardware_manifest.py # pretty-print JSON
22 + from hardware_manifest import collect_manifest # programmatic use
23 +"""
24 +
25 +from __future__ import annotations
26 +
27 +import json
28 +import platform
29 +import subprocess
30 +import sys
31 +from datetime import datetime, timezone
32 +
33 +
34 +def _run(cmd: list[str]) -> str:
35 + try:
36 + return subprocess.run(cmd, capture_output=True, text=True, timeout=30).stdout.strip()
37 + except (OSError, subprocess.TimeoutExpired):
38 + return ""
39 +
40 +
41 +def _sysctl(key: str) -> str:
42 + return _run(["sysctl", "-n", key])
43 +
44 +
45 +def _sysctl_int(key: str) -> int | None:
46 + val = _sysctl(key)
47 + try:
48 + return int(val)
49 + except ValueError:
50 + return None
51 +
52 +
53 +def _gpu_cores() -> int | None:
54 + """GPU core count via system_profiler (no sysctl key exposes it)."""
55 + out = _run(["system_profiler", "SPDisplaysDataType", "-json"])
56 + try:
57 + displays = json.loads(out)["SPDisplaysDataType"]
58 + for gpu in displays:
59 + cores = gpu.get("sppci_cores")
60 + if cores is not None:
61 + return int(cores)
62 + except (json.JSONDecodeError, KeyError, ValueError, TypeError):
63 + pass
64 + return None
65 +
66 +
67 +def _ssd_info() -> dict:
68 + out = _run(["system_profiler", "SPNVMeDataType", "-json"])
69 + try:
70 + items = json.loads(out)["SPNVMeDataType"]
71 + for controller in items:
72 + for dev in controller.get("_items", []):
73 + return {
74 + "model": dev.get("device_model", "").strip(),
75 + "size": dev.get("size", ""),
76 + "smart_status": dev.get("smart_status", ""),
77 + }
78 + except (json.JSONDecodeError, KeyError, TypeError):
79 + pass
80 + return {"model": None, "size": None, "smart_status": None}
81 +
82 +
83 +def _pkg_version(module: str) -> str | None:
84 + try:
85 + from importlib.metadata import version
86 +
87 + return version(module)
88 + except Exception:
89 + return None
90 +
91 +
92 +def _git_commit() -> dict:
93 + commit = _run(["git", "rev-parse", "HEAD"])
94 + dirty = bool(_run(["git", "status", "--porcelain"]))
95 + return {"commit": commit or None, "dirty_tree": dirty}
96 +
97 +
98 +def _thermal_state() -> str | None:
99 + # 'thermal pressure' via thermal level sysctl where available
100 + lvl = _sysctl("machdep.xcpm.cpu_thermal_level")
101 + return lvl or None
102 +
103 +
104 +def collect_manifest() -> dict:
105 + """Return the full hardware/software manifest as a dict."""
106 + mem_bytes = _sysctl_int("hw.memsize") or 0
107 + manifest = {
108 + "author": "Simon-Pierre Boucher",
109 + "contact": "contact@spboucher.ai",
110 + "project": "localvm-research",
111 + "collected_utc": datetime.now(timezone.utc).isoformat(),
112 + "chip": {
113 + "brand": _sysctl("machdep.cpu.brand_string"),
114 + "arch": platform.machine(),
115 + "cores_total": _sysctl_int("hw.ncpu"),
116 + "cores_performance": _sysctl_int("hw.perflevel0.physicalcpu"),
117 + "cores_efficiency": _sysctl_int("hw.perflevel1.physicalcpu"),
118 + "gpu_cores": _gpu_cores(),
119 + },
120 + "memory": {
121 + "unified_bytes": mem_bytes,
122 + "unified_gb": round(mem_bytes / 2**30, 1),
123 + "pagesize": _sysctl_int("hw.pagesize"),
124 + },
125 + "ssd": _ssd_info(),
126 + "os": {
127 + "product": _run(["sw_vers", "-productName"]),
128 + "version": _run(["sw_vers", "-productVersion"]),
129 + "build": _run(["sw_vers", "-buildVersion"]),
130 + "kernel": platform.release(),
131 + },
132 + "software": {
133 + "python": sys.version.split()[0],
134 + "mlx": _pkg_version("mlx"),
135 + "mlx_lm": _pkg_version("mlx-lm"),
136 + "torch": _pkg_version("torch"),
137 + "numpy": _pkg_version("numpy"),
138 + },
139 + "git": _git_commit(),
140 + "thermal_level_at_collect": _thermal_state(),
141 + }
142 + return manifest
143 +
144 +
145 +if __name__ == "__main__":
146 + print(json.dumps(collect_manifest(), indent=2))
added docs/README.md +12 −0
@@ -0,0 +1,12 @@
1 +---
2 +project: localvm-research
3 +document: docs/README
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# docs
11 +
12 +Architecture docs, diagrams, final writeups.
added experiments/micro/expA_weight_concentration/README.md +14 −0
@@ -0,0 +1,14 @@
1 +---
2 +project: localvm-research
3 +document: expA_weight_concentration/README
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# expA_weight_concentration
11 +
12 +Weight contribution concentration: can a subset of weight blocks reproduce most of each layer's output?
13 +
14 +Status: scaffolded 2026-08-11, not yet run.
added experiments/micro/expA_weight_concentration/analysis.md +12 −0
@@ -0,0 +1,12 @@
1 +---
2 +project: localvm-research
3 +document: expA_weight_concentration/analysis
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# Analysis — expA_weight_concentration
11 +
12 +*To be written after results exist. Must include the seven-field block and the evidence standard of CLAUDE.md §10.*
added experiments/micro/expA_weight_concentration/benchmark.py +33 −0
@@ -0,0 +1,33 @@
1 +# =============================================================================
2 +# Project : localvm-research
3 +# File : experiments/micro/expA_weight_concentration/benchmark.py
4 +# Purpose : Benchmark runner: Weight contribution concentration: can a subset of weight blocks reproduce most of each layer's output?
5 +# Author : Simon-Pierre Boucher
6 +# Contact : contact@spboucher.ai
7 +# Created : 2026-08-11
8 +# Modified : 2026-08-11
9 +# Platform : macOS / Apple Silicon (arm64)
10 +# License : All rights reserved (research code)
11 +# =============================================================================
12 +
13 +"""Benchmark entry point for expA_weight_concentration.
14 +
15 +Must embed the hardware manifest in all result output
16 +(see benchmarks/hardware_manifest.py) and write results to
17 +results/expA_weight_concentration/<timestamp>/.
18 +"""
19 +
20 +import sys
21 +from pathlib import Path
22 +
23 +sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "benchmarks"))
24 +from hardware_manifest import collect_manifest # noqa: E402
25 +
26 +
27 +def main() -> None:
28 + manifest = collect_manifest()
29 + raise NotImplementedError("experiment not yet implemented")
30 +
31 +
32 +if __name__ == "__main__":
33 + main()
added experiments/micro/expA_weight_concentration/hypothesis.md +33 −0
@@ -0,0 +1,33 @@
1 +---
2 +project: localvm-research
3 +document: expA_weight_concentration/hypothesis
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# Hypothesis — expA_weight_concentration
11 +
12 +```text
13 +Hypothesis
14 + <what we believe and why>
15 +
16 +Falsification criterion
17 + <the concrete measurable outcome that would prove this wrong>
18 +
19 +Method
20 + <exact procedure, model(s), data, seeds, measurement points>
21 +
22 +Baseline
23 + <what this is compared against — no straw men>
24 +
25 +Result
26 + <filled after the run: numbers, with mean/median/std and run count>
27 +
28 +Interpretation
29 + <what the numbers mean; alternative explanations considered>
30 +
31 +Next experiment
32 + <the most informative follow-up given this result>
33 +```
added experiments/micro/expB_token_stability/README.md +14 −0
@@ -0,0 +1,14 @@
1 +---
2 +project: localvm-research
3 +document: expB_token_stability/README
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# expB_token_stability
11 +
12 +Stability of important weight blocks across consecutive tokens (Jaccard, transitions, working-set lifetime)
13 +
14 +Status: scaffolded 2026-08-11, not yet run.
added experiments/micro/expB_token_stability/analysis.md +12 −0
@@ -0,0 +1,12 @@
1 +---
2 +project: localvm-research
3 +document: expB_token_stability/analysis
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# Analysis — expB_token_stability
11 +
12 +*To be written after results exist. Must include the seven-field block and the evidence standard of CLAUDE.md §10.*
added experiments/micro/expB_token_stability/benchmark.py +33 −0
@@ -0,0 +1,33 @@
1 +# =============================================================================
2 +# Project : localvm-research
3 +# File : experiments/micro/expB_token_stability/benchmark.py
4 +# Purpose : Benchmark runner: Stability of important weight blocks across consecutive tokens (Jaccard, transitions, working-set lifetime)
5 +# Author : Simon-Pierre Boucher
6 +# Contact : contact@spboucher.ai
7 +# Created : 2026-08-11
8 +# Modified : 2026-08-11
9 +# Platform : macOS / Apple Silicon (arm64)
10 +# License : All rights reserved (research code)
11 +# =============================================================================
12 +
13 +"""Benchmark entry point for expB_token_stability.
14 +
15 +Must embed the hardware manifest in all result output
16 +(see benchmarks/hardware_manifest.py) and write results to
17 +results/expB_token_stability/<timestamp>/.
18 +"""
19 +
20 +import sys
21 +from pathlib import Path
22 +
23 +sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "benchmarks"))
24 +from hardware_manifest import collect_manifest # noqa: E402
25 +
26 +
27 +def main() -> None:
28 + manifest = collect_manifest()
29 + raise NotImplementedError("experiment not yet implemented")
30 +
31 +
32 +if __name__ == "__main__":
33 + main()
added experiments/micro/expB_token_stability/hypothesis.md +33 −0
@@ -0,0 +1,33 @@
1 +---
2 +project: localvm-research
3 +document: expB_token_stability/hypothesis
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# Hypothesis — expB_token_stability
11 +
12 +```text
13 +Hypothesis
14 + <what we believe and why>
15 +
16 +Falsification criterion
17 + <the concrete measurable outcome that would prove this wrong>
18 +
19 +Method
20 + <exact procedure, model(s), data, seeds, measurement points>
21 +
22 +Baseline
23 + <what this is compared against — no straw men>
24 +
25 +Result
26 + <filled after the run: numbers, with mean/median/std and run count>
27 +
28 +Interpretation
29 + <what the numbers mean; alternative explanations considered>
30 +
31 +Next experiment
32 + <the most informative follow-up given this result>
33 +```
added experiments/micro/expC_semantic_locality/README.md +14 −0
@@ -0,0 +1,14 @@
1 +---
2 +project: localvm-research
3 +document: expC_semantic_locality/README
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# expC_semantic_locality
11 +
12 +Cross-prompt semantic locality: do same-domain prompts reuse the same weight regions?
13 +
14 +Status: scaffolded 2026-08-11, not yet run.
added experiments/micro/expC_semantic_locality/analysis.md +12 −0
@@ -0,0 +1,12 @@
1 +---
2 +project: localvm-research
3 +document: expC_semantic_locality/analysis
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# Analysis — expC_semantic_locality
11 +
12 +*To be written after results exist. Must include the seven-field block and the evidence standard of CLAUDE.md §10.*
added experiments/micro/expC_semantic_locality/benchmark.py +33 −0
@@ -0,0 +1,33 @@
1 +# =============================================================================
2 +# Project : localvm-research
3 +# File : experiments/micro/expC_semantic_locality/benchmark.py
4 +# Purpose : Benchmark runner: Cross-prompt semantic locality: do same-domain prompts reuse the same weight regions?
5 +# Author : Simon-Pierre Boucher
6 +# Contact : contact@spboucher.ai
7 +# Created : 2026-08-11
8 +# Modified : 2026-08-11
9 +# Platform : macOS / Apple Silicon (arm64)
10 +# License : All rights reserved (research code)
11 +# =============================================================================
12 +
13 +"""Benchmark entry point for expC_semantic_locality.
14 +
15 +Must embed the hardware manifest in all result output
16 +(see benchmarks/hardware_manifest.py) and write results to
17 +results/expC_semantic_locality/<timestamp>/.
18 +"""
19 +
20 +import sys
21 +from pathlib import Path
22 +
23 +sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "benchmarks"))
24 +from hardware_manifest import collect_manifest # noqa: E402
25 +
26 +
27 +def main() -> None:
28 + manifest = collect_manifest()
29 + raise NotImplementedError("experiment not yet implemented")
30 +
31 +
32 +if __name__ == "__main__":
33 + main()
added experiments/micro/expC_semantic_locality/hypothesis.md +33 −0
@@ -0,0 +1,33 @@
1 +---
2 +project: localvm-research
3 +document: expC_semantic_locality/hypothesis
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# Hypothesis — expC_semantic_locality
11 +
12 +```text
13 +Hypothesis
14 + <what we believe and why>
15 +
16 +Falsification criterion
17 + <the concrete measurable outcome that would prove this wrong>
18 +
19 +Method
20 + <exact procedure, model(s), data, seeds, measurement points>
21 +
22 +Baseline
23 + <what this is compared against — no straw men>
24 +
25 +Result
26 + <filled after the run: numbers, with mean/median/std and run count>
27 +
28 +Interpretation
29 + <what the numbers mean; alternative explanations considered>
30 +
31 +Next experiment
32 + <the most informative follow-up given this result>
33 +```
added experiments/micro/expD_progressive_reconstruction/README.md +14 −0
@@ -0,0 +1,14 @@
1 +---
2 +project: localvm-research
3 +document: expD_progressive_reconstruction/README
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# expD_progressive_reconstruction
11 +
12 +Progressive weight reconstruction: convergence of hidden-state/logit/decision error vs residual depth
13 +
14 +Status: scaffolded 2026-08-11, not yet run.
added experiments/micro/expD_progressive_reconstruction/analysis.md +12 −0
@@ -0,0 +1,12 @@
1 +---
2 +project: localvm-research
3 +document: expD_progressive_reconstruction/analysis
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# Analysis — expD_progressive_reconstruction
11 +
12 +*To be written after results exist. Must include the seven-field block and the evidence standard of CLAUDE.md §10.*
added experiments/micro/expD_progressive_reconstruction/benchmark.py +33 −0
@@ -0,0 +1,33 @@
1 +# =============================================================================
2 +# Project : localvm-research
3 +# File : experiments/micro/expD_progressive_reconstruction/benchmark.py
4 +# Purpose : Benchmark runner: Progressive weight reconstruction: convergence of hidden-state/logit/decision error vs residual depth
5 +# Author : Simon-Pierre Boucher
6 +# Contact : contact@spboucher.ai
7 +# Created : 2026-08-11
8 +# Modified : 2026-08-11
9 +# Platform : macOS / Apple Silicon (arm64)
10 +# License : All rights reserved (research code)
11 +# =============================================================================
12 +
13 +"""Benchmark entry point for expD_progressive_reconstruction.
14 +
15 +Must embed the hardware manifest in all result output
16 +(see benchmarks/hardware_manifest.py) and write results to
17 +results/expD_progressive_reconstruction/<timestamp>/.
18 +"""
19 +
20 +import sys
21 +from pathlib import Path
22 +
23 +sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "benchmarks"))
24 +from hardware_manifest import collect_manifest # noqa: E402
25 +
26 +
27 +def main() -> None:
28 + manifest = collect_manifest()
29 + raise NotImplementedError("experiment not yet implemented")
30 +
31 +
32 +if __name__ == "__main__":
33 + main()
added experiments/micro/expD_progressive_reconstruction/hypothesis.md +33 −0
@@ -0,0 +1,33 @@
1 +---
2 +project: localvm-research
3 +document: expD_progressive_reconstruction/hypothesis
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# Hypothesis — expD_progressive_reconstruction
11 +
12 +```text
13 +Hypothesis
14 + <what we believe and why>
15 +
16 +Falsification criterion
17 + <the concrete measurable outcome that would prove this wrong>
18 +
19 +Method
20 + <exact procedure, model(s), data, seeds, measurement points>
21 +
22 +Baseline
23 + <what this is compared against — no straw men>
24 +
25 +Result
26 + <filled after the run: numbers, with mean/median/std and run count>
27 +
28 +Interpretation
29 + <what the numbers mean; alternative explanations considered>
30 +
31 +Next experiment
32 + <the most informative follow-up given this result>
33 +```
added experiments/micro/expE_partial_gemm/README.md +14 −0
@@ -0,0 +1,14 @@
1 +---
2 +project: localvm-research
3 +document: expE_partial_gemm/README
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# expE_partial_gemm
11 +
12 +Partial matrix multiplication: quality vs blocks evaluated, incl. a Metal kernel bandwidth check
13 +
14 +Status: scaffolded 2026-08-11, not yet run.
added experiments/micro/expE_partial_gemm/analysis.md +12 −0
@@ -0,0 +1,12 @@
1 +---
2 +project: localvm-research
3 +document: expE_partial_gemm/analysis
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# Analysis — expE_partial_gemm
11 +
12 +*To be written after results exist. Must include the seven-field block and the evidence standard of CLAUDE.md §10.*
added experiments/micro/expE_partial_gemm/benchmark.py +33 −0
@@ -0,0 +1,33 @@
1 +# =============================================================================
2 +# Project : localvm-research
3 +# File : experiments/micro/expE_partial_gemm/benchmark.py
4 +# Purpose : Benchmark runner: Partial matrix multiplication: quality vs blocks evaluated, incl. a Metal kernel bandwidth check
5 +# Author : Simon-Pierre Boucher
6 +# Contact : contact@spboucher.ai
7 +# Created : 2026-08-11
8 +# Modified : 2026-08-11
9 +# Platform : macOS / Apple Silicon (arm64)
10 +# License : All rights reserved (research code)
11 +# =============================================================================
12 +
13 +"""Benchmark entry point for expE_partial_gemm.
14 +
15 +Must embed the hardware manifest in all result output
16 +(see benchmarks/hardware_manifest.py) and write results to
17 +results/expE_partial_gemm/<timestamp>/.
18 +"""
19 +
20 +import sys
21 +from pathlib import Path
22 +
23 +sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "benchmarks"))
24 +from hardware_manifest import collect_manifest # noqa: E402
25 +
26 +
27 +def main() -> None:
28 + manifest = collect_manifest()
29 + raise NotImplementedError("experiment not yet implemented")
30 +
31 +
32 +if __name__ == "__main__":
33 + main()
added experiments/micro/expE_partial_gemm/hypothesis.md +33 −0
@@ -0,0 +1,33 @@
1 +---
2 +project: localvm-research
3 +document: expE_partial_gemm/hypothesis
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# Hypothesis — expE_partial_gemm
11 +
12 +```text
13 +Hypothesis
14 + <what we believe and why>
15 +
16 +Falsification criterion
17 + <the concrete measurable outcome that would prove this wrong>
18 +
19 +Method
20 + <exact procedure, model(s), data, seeds, measurement points>
21 +
22 +Baseline
23 + <what this is compared against — no straw men>
24 +
25 +Result
26 + <filled after the run: numbers, with mean/median/std and run count>
27 +
28 +Interpretation
29 + <what the numbers mean; alternative explanations considered>
30 +
31 +Next experiment
32 + <the most informative follow-up given this result>
33 +```
added experiments/micro/expF_error_accumulation/README.md +14 −0
@@ -0,0 +1,14 @@
1 +---
2 +project: localvm-research
3 +document: expF_error_accumulation/README
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# expF_error_accumulation
11 +
12 +Error accumulation: which layers tolerate, amplify, or recover from controlled approximation
13 +
14 +Status: scaffolded 2026-08-11, not yet run.
added experiments/micro/expF_error_accumulation/analysis.md +12 −0
@@ -0,0 +1,12 @@
1 +---
2 +project: localvm-research
3 +document: expF_error_accumulation/analysis
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# Analysis — expF_error_accumulation
11 +
12 +*To be written after results exist. Must include the seven-field block and the evidence standard of CLAUDE.md §10.*
added experiments/micro/expF_error_accumulation/benchmark.py +33 −0
@@ -0,0 +1,33 @@
1 +# =============================================================================
2 +# Project : localvm-research
3 +# File : experiments/micro/expF_error_accumulation/benchmark.py
4 +# Purpose : Benchmark runner: Error accumulation: which layers tolerate, amplify, or recover from controlled approximation
5 +# Author : Simon-Pierre Boucher
6 +# Contact : contact@spboucher.ai
7 +# Created : 2026-08-11
8 +# Modified : 2026-08-11
9 +# Platform : macOS / Apple Silicon (arm64)
10 +# License : All rights reserved (research code)
11 +# =============================================================================
12 +
13 +"""Benchmark entry point for expF_error_accumulation.
14 +
15 +Must embed the hardware manifest in all result output
16 +(see benchmarks/hardware_manifest.py) and write results to
17 +results/expF_error_accumulation/<timestamp>/.
18 +"""
19 +
20 +import sys
21 +from pathlib import Path
22 +
23 +sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "benchmarks"))
24 +from hardware_manifest import collect_manifest # noqa: E402
25 +
26 +
27 +def main() -> None:
28 + manifest = collect_manifest()
29 + raise NotImplementedError("experiment not yet implemented")
30 +
31 +
32 +if __name__ == "__main__":
33 + main()
added experiments/micro/expF_error_accumulation/hypothesis.md +33 −0
@@ -0,0 +1,33 @@
1 +---
2 +project: localvm-research
3 +document: expF_error_accumulation/hypothesis
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# Hypothesis — expF_error_accumulation
11 +
12 +```text
13 +Hypothesis
14 + <what we believe and why>
15 +
16 +Falsification criterion
17 + <the concrete measurable outcome that would prove this wrong>
18 +
19 +Method
20 + <exact procedure, model(s), data, seeds, measurement points>
21 +
22 +Baseline
23 + <what this is compared against — no straw men>
24 +
25 +Result
26 + <filled after the run: numbers, with mean/median/std and run count>
27 +
28 +Interpretation
29 + <what the numbers mean; alternative explanations considered>
30 +
31 +Next experiment
32 + <the most informative follow-up given this result>
33 +```
added experiments/micro/expG_decision_stability/README.md +14 −0
@@ -0,0 +1,14 @@
1 +---
2 +project: localvm-research
3 +document: expG_decision_stability/README
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# expG_decision_stability
11 +
12 +Decision stability: how many token decisions are stable before full precision is available
13 +
14 +Status: scaffolded 2026-08-11, not yet run.
added experiments/micro/expG_decision_stability/analysis.md +12 −0
@@ -0,0 +1,12 @@
1 +---
2 +project: localvm-research
3 +document: expG_decision_stability/analysis
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# Analysis — expG_decision_stability
11 +
12 +*To be written after results exist. Must include the seven-field block and the evidence standard of CLAUDE.md §10.*
added experiments/micro/expG_decision_stability/benchmark.py +33 −0
@@ -0,0 +1,33 @@
1 +# =============================================================================
2 +# Project : localvm-research
3 +# File : experiments/micro/expG_decision_stability/benchmark.py
4 +# Purpose : Benchmark runner: Decision stability: how many token decisions are stable before full precision is available
5 +# Author : Simon-Pierre Boucher
6 +# Contact : contact@spboucher.ai
7 +# Created : 2026-08-11
8 +# Modified : 2026-08-11
9 +# Platform : macOS / Apple Silicon (arm64)
10 +# License : All rights reserved (research code)
11 +# =============================================================================
12 +
13 +"""Benchmark entry point for expG_decision_stability.
14 +
15 +Must embed the hardware manifest in all result output
16 +(see benchmarks/hardware_manifest.py) and write results to
17 +results/expG_decision_stability/<timestamp>/.
18 +"""
19 +
20 +import sys
21 +from pathlib import Path
22 +
23 +sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "benchmarks"))
24 +from hardware_manifest import collect_manifest # noqa: E402
25 +
26 +
27 +def main() -> None:
28 + manifest = collect_manifest()
29 + raise NotImplementedError("experiment not yet implemented")
30 +
31 +
32 +if __name__ == "__main__":
33 + main()
added experiments/micro/expG_decision_stability/hypothesis.md +33 −0
@@ -0,0 +1,33 @@
1 +---
2 +project: localvm-research
3 +document: expG_decision_stability/hypothesis
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# Hypothesis — expG_decision_stability
11 +
12 +```text
13 +Hypothesis
14 + <what we believe and why>
15 +
16 +Falsification criterion
17 + <the concrete measurable outcome that would prove this wrong>
18 +
19 +Method
20 + <exact procedure, model(s), data, seeds, measurement points>
21 +
22 +Baseline
23 + <what this is compared against — no straw men>
24 +
25 +Result
26 + <filled after the run: numbers, with mean/median/std and run count>
27 +
28 +Interpretation
29 + <what the numbers mean; alternative explanations considered>
30 +
31 +Next experiment
32 + <the most informative follow-up given this result>
33 +```
added experiments/micro/expH_ssd_feasibility/README.md +14 −0
@@ -0,0 +1,14 @@
1 +---
2 +project: localvm-research
3 +document: expH_ssd_feasibility/README
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# expH_ssd_feasibility
11 +
12 +SSD feasibility on macOS/APFS: measured random/sequential reads, F_NOCACHE, concurrent Metal compute
13 +
14 +Status: scaffolded 2026-08-11, not yet run.
added experiments/micro/expH_ssd_feasibility/analysis.md +12 −0
@@ -0,0 +1,12 @@
1 +---
2 +project: localvm-research
3 +document: expH_ssd_feasibility/analysis
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# Analysis — expH_ssd_feasibility
11 +
12 +*To be written after results exist. Must include the seven-field block and the evidence standard of CLAUDE.md §10.*
added experiments/micro/expH_ssd_feasibility/benchmark.py +33 −0
@@ -0,0 +1,33 @@
1 +# =============================================================================
2 +# Project : localvm-research
3 +# File : experiments/micro/expH_ssd_feasibility/benchmark.py
4 +# Purpose : Benchmark runner: SSD feasibility on macOS/APFS: measured random/sequential reads, F_NOCACHE, concurrent Metal compute
5 +# Author : Simon-Pierre Boucher
6 +# Contact : contact@spboucher.ai
7 +# Created : 2026-08-11
8 +# Modified : 2026-08-11
9 +# Platform : macOS / Apple Silicon (arm64)
10 +# License : All rights reserved (research code)
11 +# =============================================================================
12 +
13 +"""Benchmark entry point for expH_ssd_feasibility.
14 +
15 +Must embed the hardware manifest in all result output
16 +(see benchmarks/hardware_manifest.py) and write results to
17 +results/expH_ssd_feasibility/<timestamp>/.
18 +"""
19 +
20 +import sys
21 +from pathlib import Path
22 +
23 +sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "benchmarks"))
24 +from hardware_manifest import collect_manifest # noqa: E402
25 +
26 +
27 +def main() -> None:
28 + manifest = collect_manifest()
29 + raise NotImplementedError("experiment not yet implemented")
30 +
31 +
32 +if __name__ == "__main__":
33 + main()
added experiments/micro/expH_ssd_feasibility/hypothesis.md +33 −0
@@ -0,0 +1,33 @@
1 +---
2 +project: localvm-research
3 +document: expH_ssd_feasibility/hypothesis
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# Hypothesis — expH_ssd_feasibility
11 +
12 +```text
13 +Hypothesis
14 + <what we believe and why>
15 +
16 +Falsification criterion
17 + <the concrete measurable outcome that would prove this wrong>
18 +
19 +Method
20 + <exact procedure, model(s), data, seeds, measurement points>
21 +
22 +Baseline
23 + <what this is compared against — no straw men>
24 +
25 +Result
26 + <filled after the run: numbers, with mean/median/std and run count>
27 +
28 +Interpretation
29 + <what the numbers mean; alternative explanations considered>
30 +
31 +Next experiment
32 + <the most informative follow-up given this result>
33 +```
added pyproject.toml +39 −0
@@ -0,0 +1,39 @@
1 +# =============================================================================
2 +# Project : localvm-research
3 +# File : pyproject.toml
4 +# Purpose : Python project configuration (deps, ruff, pytest)
5 +# Author : Simon-Pierre Boucher
6 +# Contact : contact@spboucher.ai
7 +# Created : 2026-08-11
8 +# Modified : 2026-08-11
9 +# Platform : macOS / Apple Silicon (arm64)
10 +# License : All rights reserved (research code)
11 +# =============================================================================
12 +
13 +[project]
14 +name = "localvm"
15 +version = "0.0.1"
16 +description = "Research: out-of-core execution of large pretrained LLMs on consumer Apple Silicon"
17 +authors = [{ name = "Simon-Pierre Boucher", email = "contact@spboucher.ai" }]
18 +requires-python = ">=3.11"
19 +dependencies = [
20 + "numpy",
21 +]
22 +
23 +[project.optional-dependencies]
24 +dev = ["ruff", "pytest"]
25 +ml = ["mlx", "mlx-lm", "torch", "safetensors", "transformers"]
26 +
27 +[tool.setuptools.packages.find]
28 +where = ["src"]
29 +
30 +[tool.ruff]
31 +line-length = 100
32 +target-version = "py311"
33 +
34 +[tool.ruff.lint]
35 +select = ["E", "F", "W", "I", "UP", "B"]
36 +
37 +[tool.pytest.ini_options]
38 +testpaths = ["src", "experiments"]
39 +python_files = ["test_*.py"]
added research/LOG.md +29 −0
@@ -0,0 +1,29 @@
1 +---
2 +project: localvm-research
3 +document: research/LOG (append-only research log)
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# Research log (append-only)
11 +
12 +Format per entry: date/time (local, with timezone) · question · experiment · result · interpretation · decision.
13 +
14 +---
15 +
16 +## 2026-08-11 23:40 EDT — Project bootstrap
17 +
18 +- **Question:** none (administrative).
19 +- **Experiment:** none.
20 +- **Result:** Repository initialized (git, `main`). Skeleton per charter §3 created.
21 + Mandatory tooling written and functional: `tools/check_headers.py` (§0.1 enforcement),
22 + `tools/new_experiment.py` (experiment scaffolding with seven-field hypothesis block),
23 + `benchmarks/hardware_manifest.py` (macOS-native fingerprinting).
24 +- **Host hardware discovered:** Apple M5 Max, 48 GB unified memory, macOS 27.0 (build 26A5388g)
25 + — within the charter's target class (16–64 GB Apple Silicon).
26 +- **Interpretation:** n/a.
27 +- **Decision:** Begin Phase 1 (ultra-deep literature research, charter §4) immediately,
28 + fanning out across the ten mandated areas (§4.1–§4.10). Deliverables: per-topic notes in
29 + `research/notes/`, every source logged in `research/bibliography.md` with URL and access date.
added research/bibliography.md +15 −0
@@ -0,0 +1,15 @@
1 +---
2 +project: localvm-research
3 +document: research/bibliography
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# Bibliography
11 +
12 +Every consulted source, with URL and access date. Grouped by theme (mirrors `research/notes/`).
13 +Entries are appended as Phase 1 progresses; nothing is deleted.
14 +
15 +*(Populated during Phase 1 — see per-theme sections below as they are added.)*
added research/notes/README.md +12 −0
@@ -0,0 +1,12 @@
1 +---
2 +project: localvm-research
3 +document: research/notes/README
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# research/notes
11 +
12 +Per-topic Phase 1 reading notes (one file per theme).
added results/README.md +12 −0
@@ -0,0 +1,12 @@
1 +---
2 +project: localvm-research
3 +document: results/README
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# results
11 +
12 +Raw and aggregated results, results/<experiment_id>/<timestamp>/ (generated files exempt from headers).
added src/kernels/README.md +12 −0
@@ -0,0 +1,12 @@
1 +---
2 +project: localvm-research
3 +document: src/kernels/README
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-11
7 +status: draft
8 +---
9 +
10 +# src/kernels
11 +
12 +Custom Metal (.metal) and C++ kernels.
added src/localvm/__init__.py +12 −0
@@ -0,0 +1,12 @@
1 +# =============================================================================
2 +# Project : localvm-research
3 +# File : src/localvm/__init__.py
4 +# Purpose : localvm core library package root
5 +# Author : Simon-Pierre Boucher
6 +# Contact : contact@spboucher.ai
7 +# Created : 2026-08-11
8 +# Modified : 2026-08-11
9 +# Platform : macOS / Apple Silicon (arm64)
10 +# License : All rights reserved (research code)
11 +# =============================================================================
12 +"""localvm core library package root."""
added src/localvm/backends/__init__.py +12 −0
@@ -0,0 +1,12 @@
1 +# =============================================================================
2 +# Project : localvm-research
3 +# File : src/localvm/backends/__init__.py
4 +# Purpose : Compute backends package
5 +# Author : Simon-Pierre Boucher
6 +# Contact : contact@spboucher.ai
7 +# Created : 2026-08-11
8 +# Modified : 2026-08-11
9 +# Platform : macOS / Apple Silicon (arm64)
10 +# License : All rights reserved (research code)
11 +# =============================================================================
12 +"""Compute backends package."""
added src/localvm/backends/cpu_backend/__init__.py +12 −0
@@ -0,0 +1,12 @@
1 +# =============================================================================
2 +# Project : localvm-research
3 +# File : src/localvm/backends/cpu_backend/__init__.py
4 +# Purpose : Accelerate / NEON CPU compute path
5 +# Author : Simon-Pierre Boucher
6 +# Contact : contact@spboucher.ai
7 +# Created : 2026-08-11
8 +# Modified : 2026-08-11
9 +# Platform : macOS / Apple Silicon (arm64)
10 +# License : All rights reserved (research code)
11 +# =============================================================================
12 +"""Accelerate / NEON CPU compute path."""
added src/localvm/backends/cuda_optional/__init__.py +12 −0
@@ -0,0 +1,12 @@
1 +# =============================================================================
2 +# Project : localvm-research
3 +# File : src/localvm/backends/cuda_optional/__init__.py
4 +# Purpose : Isolated optional CUDA validation path (never a core dependency)
5 +# Author : Simon-Pierre Boucher
6 +# Contact : contact@spboucher.ai
7 +# Created : 2026-08-11
8 +# Modified : 2026-08-11
9 +# Platform : macOS / Apple Silicon (arm64)
10 +# License : All rights reserved (research code)
11 +# =============================================================================
12 +"""Isolated optional CUDA validation path (never a core dependency)."""
added src/localvm/backends/mlx_backend/__init__.py +12 −0
@@ -0,0 +1,12 @@
1 +# =============================================================================
2 +# Project : localvm-research
3 +# File : src/localvm/backends/mlx_backend/__init__.py
4 +# Purpose : MLX / Metal primary compute path
5 +# Author : Simon-Pierre Boucher
6 +# Contact : contact@spboucher.ai
7 +# Created : 2026-08-11
8 +# Modified : 2026-08-11
9 +# Platform : macOS / Apple Silicon (arm64)
10 +# License : All rights reserved (research code)
11 +# =============================================================================
12 +"""MLX / Metal primary compute path."""
added src/localvm/backends/mps_backend/__init__.py +12 −0
@@ -0,0 +1,12 @@
1 +# =============================================================================
2 +# Project : localvm-research
3 +# File : src/localvm/backends/mps_backend/__init__.py
4 +# Purpose : PyTorch-MPS compute path
5 +# Author : Simon-Pierre Boucher
6 +# Contact : contact@spboucher.ai
7 +# Created : 2026-08-11
8 +# Modified : 2026-08-11
9 +# Platform : macOS / Apple Silicon (arm64)
10 +# License : All rights reserved (research code)
11 +# =============================================================================
12 +"""PyTorch-MPS compute path."""
added src/localvm/instrumentation/__init__.py +12 −0
@@ -0,0 +1,12 @@
1 +# =============================================================================
2 +# Project : localvm-research
3 +# File : src/localvm/instrumentation/__init__.py
4 +# Purpose : macOS-native measurement (mach task_info, vm_stat, fs_usage, powermetrics)
5 +# Author : Simon-Pierre Boucher
6 +# Contact : contact@spboucher.ai
7 +# Created : 2026-08-11
8 +# Modified : 2026-08-11
9 +# Platform : macOS / Apple Silicon (arm64)
10 +# License : All rights reserved (research code)
11 +# =============================================================================
12 +"""macOS-native measurement (mach task_info, vm_stat, fs_usage, powermetrics)."""
added src/localvm/models/__init__.py +12 −0
@@ -0,0 +1,12 @@
1 +# =============================================================================
2 +# Project : localvm-research
3 +# File : src/localvm/models/__init__.py
4 +# Purpose : Model loading and checkpoint parsing (safetensors/GGUF)
5 +# Author : Simon-Pierre Boucher
6 +# Contact : contact@spboucher.ai
7 +# Created : 2026-08-11
8 +# Modified : 2026-08-11
9 +# Platform : macOS / Apple Silicon (arm64)
10 +# License : All rights reserved (research code)
11 +# =============================================================================
12 +"""Model loading and checkpoint parsing (safetensors/GGUF)."""
added src/localvm/quality/__init__.py +12 −0
@@ -0,0 +1,12 @@
1 +# =============================================================================
2 +# Project : localvm-research
3 +# File : src/localvm/quality/__init__.py
4 +# Purpose : Quality metrics: perplexity, KL divergence, decision stability
5 +# Author : Simon-Pierre Boucher
6 +# Contact : contact@spboucher.ai
7 +# Created : 2026-08-11
8 +# Modified : 2026-08-11
9 +# Platform : macOS / Apple Silicon (arm64)
10 +# License : All rights reserved (research code)
11 +# =============================================================================
12 +"""Quality metrics: perplexity, KL divergence, decision stability."""
added src/localvm/runtime/__init__.py +12 −0
@@ -0,0 +1,12 @@
1 +# =============================================================================
2 +# Project : localvm-research
3 +# File : src/localvm/runtime/__init__.py
4 +# Purpose : Runtime: paging, scheduling, caching, prediction
5 +# Author : Simon-Pierre Boucher
6 +# Contact : contact@spboucher.ai
7 +# Created : 2026-08-11
8 +# Modified : 2026-08-11
9 +# Platform : macOS / Apple Silicon (arm64)
10 +# License : All rights reserved (research code)
11 +# =============================================================================
12 +"""Runtime: paging, scheduling, caching, prediction."""
added src/localvm/transforms/__init__.py +12 −0
@@ -0,0 +1,12 @@
1 +# =============================================================================
2 +# Project : localvm-research
3 +# File : src/localvm/transforms/__init__.py
4 +# Purpose : Post-training transforms: quantization, decomposition, encodings
5 +# Author : Simon-Pierre Boucher
6 +# Contact : contact@spboucher.ai
7 +# Created : 2026-08-11
8 +# Modified : 2026-08-11
9 +# Platform : macOS / Apple Silicon (arm64)
10 +# License : All rights reserved (research code)
11 +# =============================================================================
12 +"""Post-training transforms: quantization, decomposition, encodings."""
added tools/check_headers.py +143 −0
@@ -0,0 +1,143 @@
1 +#!/usr/bin/env python3
2 +# =============================================================================
3 +# Project : localvm-research
4 +# File : tools/check_headers.py
5 +# Purpose : CI-style enforcement of the mandatory author header (CLAUDE.md §0.1)
6 +# Author : Simon-Pierre Boucher
7 +# Contact : contact@spboucher.ai
8 +# Created : 2026-08-11
9 +# Modified : 2026-08-11
10 +# Platform : macOS / Apple Silicon (arm64)
11 +# License : All rights reserved (research code)
12 +# =============================================================================
13 +"""Fail (exit 1) if any tracked source file lacks a conforming author header.
14 +
15 +Usage:
16 + python3 tools/check_headers.py # check all git-tracked files
17 + python3 tools/check_headers.py FILE... # check specific files
18 +
19 +Rules enforced (see CLAUDE.md §0.1):
20 + * Comment-style sources (.py .sh .zsh .yaml .yml .toml .cff Makefile
21 + CMakeLists.txt) must contain the '#'-style header block near the top.
22 + * C-family sources (.c .cpp .h .hpp .metal .swift .m .mm) must contain the
23 + '//'-style header block near the top.
24 + * Markdown documents must begin with YAML front matter declaring
25 + project/author/contact.
26 + * A shebang line may precede the header.
27 +
28 +Exemptions: generated results under results/, LICENSE, .gitignore is checked
29 +(it supports comments), CLAUDE.md (the charter predates the convention and is
30 +the specification itself).
31 +"""
32 +
33 +from __future__ import annotations
34 +
35 +import subprocess
36 +import sys
37 +from pathlib import Path
38 +
39 +REPO_ROOT = Path(__file__).resolve().parent.parent
40 +
41 +REQUIRED_FIELDS = ("Project", "File", "Purpose", "Author", "Contact",
42 + "Created", "Modified", "Platform", "License")
43 +AUTHOR = "Simon-Pierre Boucher"
44 +CONTACT = "contact@spboucher.ai"
45 +
46 +HASH_EXTS = {".py", ".sh", ".zsh", ".bash", ".yaml", ".yml", ".toml", ".cff"}
47 +SLASH_EXTS = {".c", ".cc", ".cpp", ".h", ".hpp", ".metal", ".swift", ".m", ".mm"}
48 +HASH_NAMES = {"Makefile", "CMakeLists.txt", ".gitignore"}
49 +
50 +EXEMPT_NAMES = {"LICENSE", "CLAUDE.md", "MEMORY.md"}
51 +EXEMPT_DIRS = {"results"}
52 +# How many leading lines to scan for the header block (allows shebang etc.).
53 +SCAN_LINES = 20
54 +
55 +
56 +def tracked_files() -> list[Path]:
57 + out = subprocess.run(
58 + ["git", "ls-files"], cwd=REPO_ROOT, capture_output=True, text=True, check=True
59 + ).stdout
60 + return [REPO_ROOT / line for line in out.splitlines() if line.strip()]
61 +
62 +
63 +def is_exempt(path: Path) -> bool:
64 + rel = path.relative_to(REPO_ROOT)
65 + if rel.name in EXEMPT_NAMES:
66 + return True
67 + return bool(rel.parts and rel.parts[0] in EXEMPT_DIRS)
68 +
69 +
70 +def check_comment_header(lines: list[str], prefix: str) -> list[str]:
71 + """Check for a comment-style header with all required fields near the top."""
72 + head = "\n".join(lines[:SCAN_LINES])
73 + errors = []
74 + for field in REQUIRED_FIELDS:
75 + if f"{prefix} {field}" not in head and f"{prefix} {field}" not in head:
76 + errors.append(f"missing header field: {field}")
77 + if AUTHOR not in head:
78 + errors.append(f"missing author name '{AUTHOR}'")
79 + if CONTACT not in head:
80 + errors.append(f"missing contact '{CONTACT}'")
81 + return errors
82 +
83 +
84 +def check_markdown_front_matter(lines: list[str]) -> list[str]:
85 + if not lines or lines[0].strip() != "---":
86 + return ["markdown file must start with YAML front matter (---)"]
87 + errors = []
88 + try:
89 + end = next(i for i in range(1, min(len(lines), SCAN_LINES)) if lines[i].strip() == "---")
90 + except StopIteration:
91 + return ["unterminated YAML front matter"]
92 + block = "\n".join(lines[1:end])
93 + for key in ("project: localvm-research", f"author: {AUTHOR}", f"contact: {CONTACT}"):
94 + if key not in block:
95 + errors.append(f"front matter missing '{key}'")
96 + return errors
97 +
98 +
99 +def check_file(path: Path) -> list[str]:
100 + try:
101 + text = path.read_text(encoding="utf-8", errors="replace")
102 + except OSError as exc:
103 + return [f"unreadable: {exc}"]
104 + lines = text.splitlines()
105 + if lines and lines[0].startswith("#!"):
106 + lines = lines[1:]
107 +
108 + name, ext = path.name, path.suffix
109 + if ext in HASH_EXTS or name in HASH_NAMES:
110 + return check_comment_header(lines, "#")
111 + if ext in SLASH_EXTS:
112 + return check_comment_header(lines, "//")
113 + if ext == ".md":
114 + return check_markdown_front_matter(lines)
115 + return [] # other file types are not subject to the header rule
116 +
117 +
118 +def main(argv: list[str]) -> int:
119 + paths = [Path(p).resolve() for p in argv] if argv else tracked_files()
120 + failures: dict[str, list[str]] = {}
121 + checked = 0
122 + for path in paths:
123 + if not path.is_file() or is_exempt(path):
124 + continue
125 + errors = check_file(path)
126 + if path.suffix in HASH_EXTS | SLASH_EXTS | {".md"} or path.name in HASH_NAMES:
127 + checked += 1
128 + if errors:
129 + failures[str(path.relative_to(REPO_ROOT))] = errors
130 +
131 + if failures:
132 + print(f"HEADER CHECK FAILED — {len(failures)} non-conforming file(s):\n")
133 + for rel, errors in sorted(failures.items()):
134 + print(f" {rel}")
135 + for err in errors:
136 + print(f" - {err}")
137 + return 1
138 + print(f"Header check passed ({checked} files checked).")
139 + return 0
140 +
141 +
142 +if __name__ == "__main__":
143 + sys.exit(main(sys.argv[1:]))
added tools/new_experiment.py +148 −0
@@ -0,0 +1,148 @@
1 +#!/usr/bin/env python3
2 +# =============================================================================
3 +# Project : localvm-research
4 +# File : tools/new_experiment.py
5 +# Purpose : Scaffold a header-compliant experiment directory (CLAUDE.md §3, §10)
6 +# Author : Simon-Pierre Boucher
7 +# Contact : contact@spboucher.ai
8 +# Created : 2026-08-11
9 +# Modified : 2026-08-11
10 +# Platform : macOS / Apple Silicon (arm64)
11 +# License : All rights reserved (research code)
12 +# =============================================================================
13 +"""Scaffold a new experiment directory with the mandatory structure.
14 +
15 +Creates: README.md, hypothesis.md (seven-field scientific block), benchmark.py,
16 +implementation/, results/, analysis.md — all with conforming author headers.
17 +
18 +Usage:
19 + python3 tools/new_experiment.py experiments/micro/expX_name "One-line purpose"
20 + python3 tools/new_experiment.py experiments/candidate_04 "Candidate: ..."
21 +"""
22 +
23 +from __future__ import annotations
24 +
25 +import sys
26 +from datetime import date
27 +from pathlib import Path
28 +
29 +REPO_ROOT = Path(__file__).resolve().parent.parent
30 +TODAY = date.today().isoformat()
31 +
32 +PY_HEADER = """\
33 +# =============================================================================
34 +# Project : localvm-research
35 +# File : {rel}
36 +# Purpose : {purpose}
37 +# Author : Simon-Pierre Boucher
38 +# Contact : contact@spboucher.ai
39 +# Created : {today}
40 +# Modified : {today}
41 +# Platform : macOS / Apple Silicon (arm64)
42 +# License : All rights reserved (research code)
43 +# =============================================================================
44 +"""
45 +
46 +MD_HEADER = """\
47 +---
48 +project: localvm-research
49 +document: {doc}
50 +author: Simon-Pierre Boucher
51 +contact: contact@spboucher.ai
52 +created: {today}
53 +status: draft
54 +---
55 +"""
56 +
57 +HYPOTHESIS_BODY = """
58 +# Hypothesis — {name}
59 +
60 +```text
61 +Hypothesis
62 + <what we believe and why>
63 +
64 +Falsification criterion
65 + <the concrete measurable outcome that would prove this wrong>
66 +
67 +Method
68 + <exact procedure, model(s), data, seeds, measurement points>
69 +
70 +Baseline
71 + <what this is compared against — no straw men>
72 +
73 +Result
74 + <filled after the run: numbers, with mean/median/std and run count>
75 +
76 +Interpretation
77 + <what the numbers mean; alternative explanations considered>
78 +
79 +Next experiment
80 + <the most informative follow-up given this result>
81 +```
82 +"""
83 +
84 +BENCHMARK_BODY = '''
85 +"""Benchmark entry point for {name}.
86 +
87 +Must embed the hardware manifest in all result output
88 +(see benchmarks/hardware_manifest.py) and write results to
89 +results/{name}/<timestamp>/.
90 +"""
91 +
92 +import sys
93 +from pathlib import Path
94 +
95 +sys.path.insert(0, str(Path(__file__).resolve().parents[{depth}] / "benchmarks"))
96 +from hardware_manifest import collect_manifest # noqa: E402
97 +
98 +
99 +def main() -> None:
100 + manifest = collect_manifest()
101 + raise NotImplementedError("experiment not yet implemented")
102 +
103 +
104 +if __name__ == "__main__":
105 + main()
106 +'''
107 +
108 +
109 +def scaffold(exp_dir: Path, purpose: str) -> None:
110 + if exp_dir.exists() and any(exp_dir.iterdir()):
111 + sys.exit(f"error: {exp_dir} already exists and is not empty")
112 + name = exp_dir.name
113 + rel = exp_dir.relative_to(REPO_ROOT)
114 + (exp_dir / "implementation").mkdir(parents=True, exist_ok=True)
115 + (exp_dir / "results").mkdir(exist_ok=True)
116 +
117 + def md(doc: str) -> str:
118 + return MD_HEADER.format(doc=doc, today=TODAY)
119 +
120 + (exp_dir / "README.md").write_text(
121 + md(f"{name}/README") + f"\n# {name}\n\n{purpose}\n\nStatus: scaffolded {TODAY}, not yet run.\n"
122 + )
123 + (exp_dir / "hypothesis.md").write_text(
124 + md(f"{name}/hypothesis") + HYPOTHESIS_BODY.format(name=name)
125 + )
126 + (exp_dir / "analysis.md").write_text(
127 + md(f"{name}/analysis") + f"\n# Analysis — {name}\n\n*To be written after results exist. "
128 + "Must include the seven-field block and the evidence standard of CLAUDE.md §10.*\n"
129 + )
130 + depth = len(rel.parts) # parents[] index up to repo root
131 + (exp_dir / "benchmark.py").write_text(
132 + PY_HEADER.format(rel=rel / "benchmark.py", purpose=f"Benchmark runner: {purpose}", today=TODAY)
133 + + BENCHMARK_BODY.format(name=name, depth=depth)
134 + )
135 + print(f"scaffolded {rel} ({purpose})")
136 +
137 +
138 +def main() -> None:
139 + if len(sys.argv) < 3:
140 + sys.exit(__doc__)
141 + exp_dir = (REPO_ROOT / sys.argv[1]).resolve()
142 + if REPO_ROOT not in exp_dir.parents:
143 + sys.exit("error: experiment directory must live inside the repository")
144 + scaffold(exp_dir, sys.argv[2])
145 +
146 +
147 +if __name__ == "__main__":
148 + main()
149