spb/localvm-research Public License

1# CLAUDE.md23**Project codename:** `localvm-research` (provisional — final name decided by evidence, not preference)4**Principal investigator / Author:** Simon-Pierre Boucher — <contact@spboucher.ai>5**Primary platform:** Apple Silicon Mac (macOS 14+)6**Document status:** Living research charter. Claude must treat this as the authoritative project specification.78---910## 0. Administrative conventions (MANDATORY — read before anything else)1112### 0.1 Author header requirement1314**Every single source file created in this project — without exception — must begin with a standardized author header.**1516This applies to: Python, C++, Objective-C, Swift, Metal shaders, shell scripts, Makefiles, CMake files, configuration files that support comments, and benchmark scripts.1718#### Python / shell / YAML / TOML header1920```python21# =============================================================================22#  Project   : localvm-research23#  File      : <relative/path/to/file.py>24#  Purpose   : <one-line description of what this file does>25#  Author    : Simon-Pierre Boucher26#  Contact   : contact@spboucher.ai27#  Created   : <YYYY-MM-DD>28#  Modified  : <YYYY-MM-DD>29#  Platform  : macOS / Apple Silicon (arm64)30#  License   : All rights reserved (research code)31# =============================================================================32```3334#### C++ / Metal / Swift / Objective-C header3536```cpp37// ============================================================================38//  Project   : localvm-research39//  File      : <relative/path/to/file.cpp>40//  Purpose   : <one-line description>41//  Author    : Simon-Pierre Boucher42//  Contact   : contact@spboucher.ai43//  Created   : <YYYY-MM-DD>44//  Modified  : <YYYY-MM-DD>45//  Platform  : macOS / Apple Silicon (arm64) — Metal / Accelerate / MLX46//  License   : All rights reserved (research code)47// ============================================================================48```4950#### Markdown research documents header (front matter)5152```markdown53---54project: localvm-research55document: <name>56author: Simon-Pierre Boucher57contact: contact@spboucher.ai58created: <YYYY-MM-DD>59status: draft | reviewed | final60---61```6263Rules:64651. The header must be the **first content** of the file (after a shebang line if one exists).662. `Modified` must be updated whenever the file is substantially changed.673. A CI-style check script (`tools/check_headers.py`) must be written early in the project and run before every commit; it fails if any tracked source file lacks a conforming header.684. Generated files (e.g., results JSON) are exempt, but the generator must embed `"author": "Simon-Pierre Boucher", "contact": "contact@spboucher.ai"` in output metadata where format permits.6970### 0.2 macOS-first design constraint7172**Everything in this project must be designed to run on a Mac.** Specifically:7374* Target machine class: Apple Silicon (M1/M2/M3/M4 family), 16–64 GB unified memory, internal NVMe SSD.75* Default compute paths: **MLX**, **Metal (MSL kernels)**, **Accelerate/BLAS**, **PyTorch with MPS backend**, plain CPU (arm64 NEON).76* CUDA is allowed **only** as an optional, clearly isolated validation path (`src/backends/cuda_optional/`), never a dependency of the core runtime.77* All I/O experiments must account for macOS specifics: APFS behavior, the unified page cache, `mmap` semantics on macOS, `F_NOCACHE`/`fcntl` instead of Linux `O_DIRECT`, `posix_fadvise` absence, `purge`/`vm_stat` for memory pressure measurement, and Apple SSD controller characteristics.78* Instrumentation must use macOS-native sources where possible: `task_info` / `mach` APIs, `vm_stat`, `fs_usage`, `powermetrics` (energy, requires sudo — degrade gracefully), Instruments traces where practical, `sysctl hw.*` for hardware discovery.79* Build tooling: `uv` or `pip` + `venv` for Python; `cmake` + AppleClang for C++; `xcrun -sdk macosx metal` for Metal shader compilation. No Linux-only assumptions anywhere in the core.80* A hardware manifest (`benchmarks/hardware_manifest.py`) must record chip model, core counts (P/E), GPU core count, memory size, SSD model, macOS version, and software versions into every result file.8182### 0.3 Repository discipline8384* Git from day one. Meaningful commits. No giant "misc" commits.85* Every experiment result must be reproducible from: (commit hash) + (config file) + (seed) + (hardware manifest).86* Python: type hints, `ruff` for lint/format, `pytest` for correctness tests of numerical code.87* C++: `-Wall -Wextra -Werror`, sanitizers in debug builds.88* No result may be reported from an uncommitted working tree.8990---9192## 1. Mission9394Your objective is to investigate, design, implement, and experimentally validate a genuinely new way to run **existing pretrained open-weight large language models that normally do not fit comfortably in consumer-Mac memory** on ordinary local Apple Silicon hardware.9596The target problem is **not**:9798* training a new foundation model;99* designing a smaller replacement model;100* merely applying standard quantization;101* merely using CPU offload;102* merely using SSD swap;103* merely pruning a model once and accepting permanent quality loss;104* or reproducing llama.cpp, MLX, PowerInfer, FlexGen, speculative decoding, or another existing system.105106The target is broader and more ambitious:107108> **Given an already-trained model whose normal memory and bandwidth requirements exceed the target Mac, determine whether its execution can be reorganized, transformed, compiled, approximated, decomposed, scheduled, cached, paged, predicted, refined, or otherwise executed differently so that the model remains meaningfully useful on consumer Apple Silicon hardware with dramatically lower resident memory and/or memory bandwidth requirements.**109110Do not assume in advance what the solution should be.111Do not force a particular architecture.112Do not begin implementation before understanding the state of the art.113Your job is to discover what is actually possible.114115---116117## 2. Core research question118119Investigate:120121> **Can an existing dense or MoE pretrained LLM be transformed post-training into an execution representation whose instantaneous working set is dramatically smaller than the full checkpoint, while preserving most or all of the original model's useful capabilities?**122123The most important distinction is:124125```text126total model size127≠128resident model size129≠130bytes read per token131≠132parameters materially required for a particular token133```134135Explore whether these quantities can be decoupled.136137A successful system could potentially make a model much larger than available unified memory usable locally on a Mac, where additional model size primarily increases latency or storage requirements rather than making execution impossible.138139### Guiding principle140141Do not optimize first for benchmark scores. Initially optimize for discovering whether there is a **new execution regime**.142143The main scientific question is:144145```text146How little of the original model must actually be represented,147loaded, reconstructed, or evaluated to reproduce its behavior?148```149150Treat this as a systems + machine-learning research problem. You are free to discover that the initial premise is wrong. **Negative experimental results are useful.**151152---153154## 3. Project structure (real research-project layout)155156The repository must follow this layout. Create the skeleton (with header-compliant placeholder files) before Phase 1 concludes.157158```text159localvm-research/160├── CLAUDE.md                     # this charter161├── README.md                     # public-facing summary (written last, updated continuously)162├── CITATION.cff                  # citation metadata (author: Simon-Pierre Boucher)163├── LICENSE164├── pyproject.toml                # Python project config (ruff, pytest, deps)165├── Makefile                      # top-level tasks: setup, lint, test, bench, headers166│167├── research/                     # the scientific paper trail168│   ├── LOG.md                    # dated research log (append-only)169│   ├── state_of_the_art.md      # Phase 2 deliverable170│   ├── research_gaps.md         # Phase 3 deliverable171│   ├── candidate_ranking.md     # Phase 4 deliverable172│   ├── bibliography.md          # every source, with links and access dates173│   ├── novelty_check.md         # Phase 11 deliverable174│   └── notes/                    # per-topic reading notes (one file per theme)175│176├── src/                          # core library code177│   ├── localvm/178│   │   ├── __init__.py179│   │   ├── models/               # model loading, checkpoint parsing (safetensors/GGUF)180│   │   ├── transforms/           # quantization, decomposition, encodings181│   │   ├── runtime/              # paging, scheduling, caching, prediction182│   │   ├── backends/183│   │   │   ├── mlx_backend/      # MLX / Metal primary path184│   │   │   ├── mps_backend/      # PyTorch-MPS path185│   │   │   ├── cpu_backend/      # Accelerate / NEON path186│   │   │   └── cuda_optional/    # isolated, optional187│   │   ├── instrumentation/      # macOS-native measurement (mach, vm_stat, fs_usage…)188│   │   └── quality/              # perplexity, KL, decision-stability metrics189│   └── kernels/                  # custom Metal (.metal) and C++ kernels190│191├── experiments/                  # micro-experiments and candidate prototypes192│   ├── micro/193│   │   ├── expA_weight_concentration/194│   │   ├── expB_token_stability/195│   │   ├── expC_semantic_locality/196│   │   ├── expD_progressive_reconstruction/197│   │   ├── expE_partial_gemm/198│   │   ├── expF_error_accumulation/199│   │   ├── expG_decision_stability/200│   │   └── expH_ssd_feasibility/201│   ├── candidate_01/202│   ├── candidate_02/203│   └── candidate_03/204│       └── (each candidate contains:)205│           ├── README.md206│           ├── hypothesis.md207│           ├── implementation/208│           ├── benchmark.py209│           ├── results/210│           └── analysis.md211│212├── benchmarks/                   # baselines and standardized harness213│   ├── harness.py                # unified benchmark runner214│   ├── hardware_manifest.py      # macOS hardware/software fingerprinting215│   ├── baselines/                # llama.cpp, MLX, mmap, offload configs216│   └── datasets/                 # eval prompts: code, math, chat, FR, EN, reasoning217│218├── results/                      # raw + aggregated results (JSON/CSV + plots)219│   └── <experiment_id>/<timestamp>/220│221├── tools/222│   ├── check_headers.py          # enforces §0.1223│   ├── new_experiment.py         # scaffolds a compliant experiment directory224│   └── report.py                 # generates result tables/plots225│226└── docs/                         # architecture docs, diagrams, final writeups227```228229Every experiment directory is scaffolded by `tools/new_experiment.py`, which auto-inserts the author header and a `hypothesis.md` template containing the seven-field scientific-discipline block (§10).230231---232233## 4. Phase 1 — Ultra-deep web and literature research234235Before proposing an architecture, perform an extremely deep search of current literature, repositories, technical reports, conference papers, preprints, blog posts, implementations, issue discussions, benchmarks, and systems research.236237The search must be current. Search broadly and recursively. Do not stop after finding several obvious papers. Follow citations backward and forward whenever useful.238239Search arXiv, OpenReview, conference proceedings (MLSys, OSDI, SOSP, ASPLOS, ISCA, NeurIPS, ICML, ICLR, ACL), GitHub, Hugging Face, systems research venues, vendor engineering documentation (especially **Apple ML/Metal engineering material and MLX repos**), academic project pages, and relevant technical discussions.240241Prefer primary sources. For every important technique, find the actual paper and, whenever available, the implementation. Log every consulted source in `research/bibliography.md` with URL and access date.242243### Areas that must be investigated244245At minimum investigate all of the following, while remaining open to unrelated approaches.246247#### 4.1 Quantization248249PTQ; QAT; 8-bit; 6-bit; 4-bit; 3-bit; 2-bit; 1.58-bit; ternary weights; 1-bit approaches; mixed precision; per-layer precision; per-channel precision; per-token precision; dynamic precision; progressive precision; residual quantization; recurrent residual quantization; additive quantization; vector quantization; lattice quantization; codebook methods; weight-only quantization; activation quantization; KV-cache quantization; extreme low-bit inference.250251Study quality degradation and actual memory-bandwidth effects **separately**. Pay particular attention to which formats have efficient Metal/MLX kernels versus which exist only on CUDA.252253#### 4.2 Activation sparsity254255Contextual sparsity; dynamic activation sparsity; FFN sparsity; neuron activation prediction; top-k neuron selection; ReLUfication; SwiGLU sparsification; structured sparsity; unstructured sparsity; N:M sparsity; activation predictors; learned sparsity routers; post-training induced sparsity.256257Study systems including but not limited to: DejaVu; ShadowLLM; PowerInfer; PowerInfer-2; DynamicInfer; SparQ; contextual sparsity systems. Determine exactly what is already known, and what portions assume discrete-GPU architectures that do not map to unified memory.258259#### 4.3 Weight sparsity and pruning260261SparseGPT; Wanda; magnitude pruning; structured pruning; block pruning; channel pruning; layer dropping; width pruning; depth pruning; one-shot pruning; dynamic pruning; recoverable pruning; reversible pruning; conditional pruning.262263Ask whether discarded weights could instead live on the Mac's NVMe SSD and be recovered only when needed.264265#### 4.4 Out-of-core inference266267CPU/GPU offload; RAM/VRAM tiering; NVMe offload; mmap; asynchronous I/O; direct I/O; page cache behavior; pinned memory; unified memory; **Apple Silicon memory behavior specifically**; GPU prefetch; tensor paging; weight streaming; SSD-to-GPU pipelines; near-storage inference; computational storage.268269Investigate systems such as: FlexGen; DeepSpeed inference; ZeRO-Inference; llama.cpp; MLX; PowerInfer; M2Cache; SolidAttention; Petals where relevant; distributed consumer inference systems.270271Measure whether the true bottleneck is capacity, bandwidth, latency, random I/O, or synchronization — **on macOS/APFS/Apple NVMe, not on Linux assumptions**.272273#### 4.5 Model decomposition274275Research whether pretrained weights can be represented as `base + residual`, `shared component + layer-specific correction`, or `low-rank component + sparse residual`.276277Investigate: SVD; truncated SVD; tensor decomposition; low-rank factorization; LoRA-like decomposition of existing weights; cross-layer sharing; DeltaLLM; matrix dictionaries; learned codebooks; basis decomposition; Kronecker decomposition; tensor trains; product quantization; weight clustering; block-level factorization; shared latent weight representations.278279Determine whether model layers contain exploitable redundancy that existing runtimes ignore.280281#### 4.6 Progressive and approximate computation282283Progressive inference; anytime neural networks; early exit; adaptive computation; dynamic depth; conditional computation; residual refinement; coarse-to-fine inference; iterative refinement; approximate matrix multiplication; error-bounded GEMM; approximate computing; lazy tensor evaluation; partial matrix multiplication; adaptive precision numerical computing.284285Do not limit this search to LLM research. Look at computer architecture, numerical linear algebra, databases, graphics, signal processing, scientific computing, and HPC. Potentially useful concepts may already exist outside machine learning.286287#### 4.7 Speculative execution288289Speculative decoding; self-speculative decoding; draft models; Medusa-like approaches; verification methods; exact distribution preservation; optimistic execution; rollback; branch prediction; confidence-based speculative computation.290291Ask whether speculation can happen **inside a transformer forward pass**, not only across future tokens.292293#### 4.8 Memory systems (outside AI literature)294295Virtual memory; demand paging; working-set theory; page replacement; TLBs; cache associativity; hardware prefetchers; branch predictors; speculative execution; NUMA; memory compression (including **macOS compressed memory**); compressed caches; tiered memory; CXL memory; object stores; database buffer pools; columnar execution; query optimizers.296297Ask: *What ideas from operating systems and CPU architecture have never been properly translated to neural-weight execution?*298299#### 4.9 Numerical error analysis300301Interval arithmetic; affine arithmetic; probabilistic bounds; matrix norm bounds; perturbation theory; Lipschitz bounds; error propagation; certified neural networks; robustness certification; approximate linear algebra; bounds for quantized networks.302303A potentially important direction: can computation terminate when additional accuracy can no longer meaningfully affect the output? Do not assume this is feasible. Test it.304305#### 4.10 Output-decision stability306307Investigate the distinction between *reproducing exact hidden states* and *producing the same useful output*.308309For greedy decoding, investigate whether a token decision can be certified without fully reconstructing every upstream operation. For sampling, investigate whether approximate intermediate computation can still preserve or closely approximate the original probability distribution.310311Explore: top-1 logit margins; top-k stability; KL divergence; total variation distance; distributional guarantees; acceptance/rejection correction; speculative verification.312313This is an important area but is **not prescribed as the final approach**.314315---316317## 5. Phase 2 — State-of-the-art map318319Produce `research/state_of_the_art.md` (with the §0.1 front-matter header).320321It must organize every relevant technique by:322323```text324Technique325Problem addressed326Model modification required?327Retraining required?328Memory reduction329Bandwidth reduction330Compute reduction331Latency effect332Quality degradation333Hardware assumptions (explicitly: does it work on Apple Silicon / Metal?)334Open-source implementation (and whether it builds on macOS arm64)335Main limitation336Opportunity for extension337```338339Do not simply summarize papers. Identify where approaches overlap. Identify combinations that have already been tried. Identify ideas that initially seem novel but are actually already known.340341---342343## 6. Phase 3 — Identify genuine gaps344345Produce `research/research_gaps.md`.346347For every promising gap explain:3483491. what existing systems do;3502. what they do not do;3513. why the missing capability might matter;3524. whether there is a plausible mathematical or systems reason it could work;3535. the biggest reason it might fail;3546. the smallest experiment capable of falsifying it — **runnable on a Mac**.355356Generate many candidate ideas. Aim for at least **20 substantially different approaches**. Do not make them superficial variants of the same idea.357358Example categories (non-binding): execution reordering; learned weight paging; progressive reconstruction; activation-conditioned decompression; semantic caches; low-rank hot path + residual cold path; predictive SSD reads; dynamic precision; partial GEMM; temporary model specialization; token-dependent model materialization; model-state compression; weight synthesis; cross-layer reuse; hidden-state approximation; error-controlled execution; reversible approximation; adaptive layer reconstruction.359360These are examples only. Do not constrain the search to them.361362---363364## 7. Phase 4 — Rank candidate approaches365366Create `research/candidate_ranking.md`.367368Score every idea 1–10, with written reasoning, on:369370```text371Novelty372Technical plausibility373Expected memory reduction374Expected bandwidth reduction375Expected quality retention376Implementation complexity377Hardware practicality on Apple Silicon (Metal/MLX kernel feasibility, unified memory fit)378Post-training compatibility379Potential research significance380Risk381```382383Select roughly **3–5 strongest candidates** for experiments.384385Prefer ideas that could fundamentally change the scaling relationship between `model size` and `required resident memory / bytes transferred per token`.386387---388389## 8. Phase 5 — Experimental framework390391Build an experimental framework **before** attempting large models. The system must make experiments reproducible.392393Preferred languages and stacks (Mac-first):394395```text396Python (MLX, PyTorch-MPS, NumPy) for research tooling397C++ (AppleClang, Accelerate) where low-level control matters398Metal Shading Language for custom Apple GPU kernels399Swift/Objective-C shims only where mach/IOKit APIs require them400CUDA only as isolated optional validation on non-Mac hardware401```402403Do not prematurely optimize. Correctness and measurement come first.404405### 8.1 Initial model sizes406407Start small. Use models in approximately this progression:408409```text4100.5B–1B  →  3B  →  7B–8B  →  14B  →  32B411```412413Only move to 70B+ if evidence supports it. Prefer modern open-weight architectures representative of models we ultimately want to run, with checkpoints available in safetensors and GGUF.414415### 8.2 Instrumentation (macOS-native)416417The framework must measure, when applicable:418419```text420resident RAM (mach task_info)         peak RAM421mapped memory                         GPU/unified memory in use422SSD reads (fs_usage / iostat)         SSD bytes/token423RAM bytes/token                       effective weight bytes/token424tokens/second                         time-to-first-token425per-token latency                     CPU utilization (P vs E cores)426GPU utilization                       SSD utilization427energy via powermetrics (if sudo)     cache hit rate428page faults (vm_stat deltas)          number of parameters touched429number of blocks touched              effective precision used430recomputation count                   prediction hit rate431memory-pressure / compressor activity thermal state (throttling detection)432```433434Also measure quality:435436```text437perplexity                 logit correlation438KL divergence              same greedy token rate439top-k overlap              task benchmark accuracy440generation similarity      long-context behavior441coding behavior            reasoning behavior442```443444### 8.3 Most important systems metric445446Track **BYTES READ PER GENERATED TOKEN** separately from model size.447Also track **ACTIVE / TOUCHED PARAMETERS PER TOKEN** where meaningful.448449A system that stores a 150 GB checkpoint but reads only 5 GB per generated token may be much more interesting than one that compresses the checkpoint to 70 GB but reads all 70 GB every token.450451### 8.4 Baselines452453At minimum compare against appropriate configurations of:454455```text456full precision where practical457Q8 / Q6 / Q5 / Q4 / Q3 / Q2 (where practical)458llama.cpp (Metal build)459MLX (native)460CPU offloading461mmap streaming462standard SSD offloading463```464465Also reproduce relevant published methods when feasible. Do not claim improvement against straw-man baselines.466467---468469## 9. Phase 6 — Micro-experiments470471Before implementing a large runtime, answer fundamental questions empirically. Each lives in `experiments/micro/` with the standard scaffold.472473### Experiment A — Weight contribution concentration474For each transformer layer and token, measure how much of the output norm arises from subsets of weight blocks. Ask: can 10%, 20%, 40%, or 60% of blocks reproduce most of the layer output? Test across: code, math, chat, French, English, reasoning, random text, long context.475476### Experiment B — Stability across consecutive tokens477Measure overlap between important blocks at tokens t, t+1, t+2. Compute Jaccard similarity, transition probabilities, working-set lifetime. Determine whether weight access is predictable.478479### Experiment C — Cross-prompt semantic locality480Ask whether prompts belonging to the same semantic domain repeatedly rely on similar regions. If so, investigate whether domain-specific weight caches are possible.481482### Experiment D — Progressive weight reconstruction483Represent weights using multiple progressive approximations (2-bit only; 2+residual; 2+2 residuals; …). For each token/layer determine how rapidly hidden-state error, logit error, and the token decision converge.484485### Experiment E — Partial matrix multiplication486Compute only selected weight blocks. Measure quality vs blocks evaluated. Then determine whether important blocks can be predicted cheaply. Implement at least one Metal kernel variant to check that block-skipping actually saves bandwidth on Apple GPUs (not just FLOPs on paper).487488### Experiment F — Error accumulation489Introduce controlled approximation at individual layers. Determine which layers tolerate error, which amplify error, which recover from error. Map model sensitivity.490491### Experiment G — Decision stability492For an approximate forward pass, compare top logits against the exact model. Ask whether many token decisions become stable before full model precision is available.493494### Experiment H — SSD feasibility (macOS-specific)495Simulate realistic storage on the actual Mac's internal NVMe. Do NOT rely on theoretical bandwidth. Measure actual random and sequential reads at 4 KB, 16 KB, 64 KB, 256 KB, 1 MB, 4 MB — with and without page cache (`F_NOCACHE`), cold vs warm APFS state. Measure concurrent reads while Metal GPU compute occurs. Test whether useful prefetch overlap exists. Record SSD model and thermal state; Apple SSDs throttle.496497---498499## 10. Scientific discipline500501For every experiment explicitly write, in its `hypothesis.md` / `analysis.md`:502503```text504Hypothesis505Falsification criterion506Method507Baseline508Result509Interpretation510Next experiment511```512513Avoid confirmation bias. If an idea fails, record why. **Do not silently discard failures.**514515### Evidence standard516517Never write "this is faster / novel / better / should scale" without evidence. Use benchmark numbers. Whenever possible report:518519```text520mean, median, standard deviation, number of runs,521hardware (chip, RAM, SSD, macOS version), model, context length,522generation length, quantization, software versions (MLX, PyTorch, Metal),523thermal conditions524```525526---527528## 11. Phase 7 — Prototype candidate architectures529530For each of the strongest ideas create its own directory:531532```text533experiments/candidate_01/534experiments/candidate_02/535experiments/candidate_03/536```537538Each candidate must include:539540```text541README.md          hypothesis.md542implementation/    benchmark.py543results/           analysis.md544```545546All implementation files carry the §0.1 header and must run on the primary Mac target without CUDA.547548---549550## 12. Phase 8 — Automatic research loop551552Operate as a research agent. After every experiment:5535541. analyze results;5552. determine the most informative next experiment;5563. search the literature again if results reveal something unexpected;5574. update hypotheses;5585. modify the candidate architecture;5596. rerun;5607. compare;5618. continue until evidence strongly favors or rejects the approach.562563Do not follow a rigid predefined implementation roadmap if experiments contradict it.564565### Research log566567Maintain `research/LOG.md` (append-only). Every substantial action must contain:568569```text570date/time (local, with timezone)571question572experiment573result574interpretation575decision576```577578This must make the entire reasoning process auditable.579580---581582## 13. Phase 9 — Try to discover a new execution abstraction583584Do not merely seek an optimization. Seek an abstraction that could change how large local models are executed. Useful questions include:585586```text587Could model weights behave like virtual memory?588Could only a token-specific working set be materialized?589Could a model be represented as a cheap approximate core plus recoverable information?590Could computation proceed before all weights are available?591Could missing computation be added later only when needed?592Could weights be reconstructed from shared bases?593Could hidden-state trajectories predict future memory accesses?594Could the system learn its own cache policy?595Could model execution become demand-driven?596Could the runtime optimize for decision stability rather than numerical exactness?597Could a model larger than RAM become usable because RAM limits working-set size598rather than total model size?599Could Apple's unified memory + fast NVMe make an abstraction viable that600discrete-GPU architectures cannot support?601```602603These are research questions, not required design decisions.604605---606607## 14. Phase 10 — Build the best prototype supported by evidence608609Once experiments clearly favor an architecture, build a prototype runtime. A possible CLI shape could eventually be:610611```bash612localvm compile MODEL_PATH \613    --target-memory 16GB \614    --storage /Volumes/FastNVMe/localvm615```616617then:618619```bash620localvm run COMPILED_MODEL621```622623Naming and architecture should only be finalized after research.624625### Compilation stage626627If useful, compilation may perform offline operations such as: profiling; weight analysis; quantization; factorization; clustering; reordering; block creation; index construction; activation tracing; cache-profile creation; predictor training; error-bound estimation; progressive encoding; layout optimization (aligned to APFS/Metal buffer constraints).628629Compilation can take substantial time. Inference must be the optimized stage.630631### Hard constraint632633The original pretrained model must remain the source model. You may transform its representation post-training. Small auxiliary predictors, indexes, adapters, or calibration passes are allowed if justified. **Do not solve the problem by simply training a new smaller LLM to replace it.**634635---636637## 15. Target hardware638639Primary research target:640641```text642Apple Silicon Mac (laptop or desktop)64316–64 GB unified memory644internal Apple NVMe SSD (2–8 GB/s class)645Metal-capable GPU sharing memory with CPU646```647648Apple Silicon is the primary platform because:649650```text651CPU and GPU share unified memory652Metal allows custom kernels653fast internal SSDs are standard654large memory configurations exist (up to 128–192 GB on desktop parts)655memory compression is built into the OS656```657658The architecture should nevertheless remain conceptually hardware-independent; document (do not implement, unless trivial) how each mechanism would map to x86 + discrete GPU.659660### Stretch target661662```text663model total representation: 100–250+ GB664machine unified memory:     16–32 GB665useful generation speed:    interactive or near-interactive666quality:                    close to original model667```668669Do not assume this target is achievable. The research must establish the actual limits.670671---672673## 16. What counts as a breakthrough674675A result is scientifically interesting if at least one of the following is demonstrated:676677**A.** A model significantly larger than unified memory runs locally on a Mac with acceptable interactive latency.678**B.** Bytes transferred per token become substantially smaller than the compressed checkpoint size.679**C.** Only a small token-dependent fraction of model information is required during typical inference.680**D.** A progressive or conditional execution mechanism preserves model quality while avoiding large amounts of weight loading.681**E.** A new post-training representation creates a qualitatively better storage/RAM/quality tradeoff than fixed quantization.682**F.** A new caching, prediction, scheduling, decomposition, or reconstruction mechanism materially changes out-of-core inference economics.683684## 17. Failure criteria685686Be willing to conclude that an approach does not work. For example:687688```text689SSD bandwidth fundamentally dominates690weight accesses are insufficiently predictable691required working set is nearly the entire model692approximation errors compound uncontrollably693low-bit base models destroy routing information694prediction overhead exceeds savings695random I/O eliminates theoretical advantages696quality degradation is unacceptable697macOS I/O or Metal dispatch overhead erases theoretical wins698```699700If these occur, document them in `research/LOG.md` and the relevant `analysis.md`, then pivot.701702---703704## 18. Deliverables705706Eventually produce (all with compliant headers):707708```text709README.md710CITATION.cff711research/712    state_of_the_art.md713    research_gaps.md714    candidate_ranking.md715    novelty_check.md716    LOG.md717    bibliography.md718src/719experiments/720benchmarks/721results/722docs/723tools/724```725726The final README must explain:727728```text729the problem730what existing systems do731what gap was discovered732the proposed architecture733why it should work734experimental evidence (with numbers, on named Mac hardware)735performance736limitations737how to reproduce (exact Mac setup instructions)738future research739```740741---742743## 19. Phase 11 — Novelty verification744745Before claiming novelty, perform a dedicated final literature search using terminology derived from the architecture actually discovered. Search for conceptual equivalents, not merely identical terminology.746747For example, if the architecture resembles `lazy neural execution`, `progressive weight materialization`, `conditional tensor paging`, or `activation-conditioned decompression` — search every plausible synonym.748749Assume an idea is **not** novel until evidence suggests otherwise. Record the process and conclusion in `research/novelty_check.md`.750751---752753## 20. Most important instruction754755Do not become attached to any particular solution suggested in this document.756757The purpose of this project is **not to implement a preconceived WeightVM, NeuralOS, progressive quantizer, sparse runtime, or paging engine**. Those ideas are merely clues.758759The actual assignment is:760761> **Search deeply enough, reason independently enough, and experiment aggressively enough to discover the best technically plausible way of making an existing large pretrained model usable on a Mac that normally should not be able to run it.**762763Start with literature. Then generate hypotheses. Then falsify them experimentally. Let evidence determine the architecture.764765The ideal outcome is not simply another optimization. The ideal outcome is a mechanism that changes the relationship:766767```text768model size → hardware required769```770771into something closer to:772773```text774instantaneous useful working set → hardware required775```776777while keeping the intelligence already present in the original pretrained model.778779---780781*Author: Simon-Pierre Boucher — contact@spboucher.ai — All research artifacts in this repository carry this attribution.*782