# CLAUDE.md **Project codename:** `localvm-research` (provisional — final name decided by evidence, not preference) **Principal investigator / Author:** Simon-Pierre Boucher — **Primary platform:** Apple Silicon Mac (macOS 14+) **Document status:** Living research charter. Claude must treat this as the authoritative project specification. --- ## 0. Administrative conventions (MANDATORY — read before anything else) ### 0.1 Author header requirement **Every single source file created in this project — without exception — must begin with a standardized author header.** This applies to: Python, C++, Objective-C, Swift, Metal shaders, shell scripts, Makefiles, CMake files, configuration files that support comments, and benchmark scripts. #### Python / shell / YAML / TOML header ```python # ============================================================================= # Project : localvm-research # File : # Purpose : # Author : Simon-Pierre Boucher # Contact : contact@spboucher.ai # Created : # Modified : # Platform : macOS / Apple Silicon (arm64) # License : All rights reserved (research code) # ============================================================================= ``` #### C++ / Metal / Swift / Objective-C header ```cpp // ============================================================================ // Project : localvm-research // File : // Purpose : // Author : Simon-Pierre Boucher // Contact : contact@spboucher.ai // Created : // Modified : // Platform : macOS / Apple Silicon (arm64) — Metal / Accelerate / MLX // License : All rights reserved (research code) // ============================================================================ ``` #### Markdown research documents header (front matter) ```markdown --- project: localvm-research document: author: Simon-Pierre Boucher contact: contact@spboucher.ai created: status: draft | reviewed | final --- ``` Rules: 1. The header must be the **first content** of the file (after a shebang line if one exists). 2. `Modified` must be updated whenever the file is substantially changed. 3. A CI-style check script (`tools/check_headers.py`) must be written early in the project and run before every commit; it fails if any tracked source file lacks a conforming header. 4. Generated files (e.g., results JSON) are exempt, but the generator must embed `"author": "Simon-Pierre Boucher", "contact": "contact@spboucher.ai"` in output metadata where format permits. ### 0.2 macOS-first design constraint **Everything in this project must be designed to run on a Mac.** Specifically: * Target machine class: Apple Silicon (M1/M2/M3/M4 family), 16–64 GB unified memory, internal NVMe SSD. * Default compute paths: **MLX**, **Metal (MSL kernels)**, **Accelerate/BLAS**, **PyTorch with MPS backend**, plain CPU (arm64 NEON). * CUDA is allowed **only** as an optional, clearly isolated validation path (`src/backends/cuda_optional/`), never a dependency of the core runtime. * All I/O experiments must account for macOS specifics: APFS behavior, the unified page cache, `mmap` semantics on macOS, `F_NOCACHE`/`fcntl` instead of Linux `O_DIRECT`, `posix_fadvise` absence, `purge`/`vm_stat` for memory pressure measurement, and Apple SSD controller characteristics. * Instrumentation must use macOS-native sources where possible: `task_info` / `mach` APIs, `vm_stat`, `fs_usage`, `powermetrics` (energy, requires sudo — degrade gracefully), Instruments traces where practical, `sysctl hw.*` for hardware discovery. * Build tooling: `uv` or `pip` + `venv` for Python; `cmake` + AppleClang for C++; `xcrun -sdk macosx metal` for Metal shader compilation. No Linux-only assumptions anywhere in the core. * A hardware manifest (`benchmarks/hardware_manifest.py`) must record chip model, core counts (P/E), GPU core count, memory size, SSD model, macOS version, and software versions into every result file. ### 0.3 Repository discipline * Git from day one. Meaningful commits. No giant "misc" commits. * Every experiment result must be reproducible from: (commit hash) + (config file) + (seed) + (hardware manifest). * Python: type hints, `ruff` for lint/format, `pytest` for correctness tests of numerical code. * C++: `-Wall -Wextra -Werror`, sanitizers in debug builds. * No result may be reported from an uncommitted working tree. --- ## 1. Mission Your objective is to investigate, design, implement, and experimentally validate a genuinely new way to run **existing pretrained open-weight large language models that normally do not fit comfortably in consumer-Mac memory** on ordinary local Apple Silicon hardware. The target problem is **not**: * training a new foundation model; * designing a smaller replacement model; * merely applying standard quantization; * merely using CPU offload; * merely using SSD swap; * merely pruning a model once and accepting permanent quality loss; * or reproducing llama.cpp, MLX, PowerInfer, FlexGen, speculative decoding, or another existing system. The target is broader and more ambitious: > **Given an already-trained model whose normal memory and bandwidth requirements exceed the target Mac, determine whether its execution can be reorganized, transformed, compiled, approximated, decomposed, scheduled, cached, paged, predicted, refined, or otherwise executed differently so that the model remains meaningfully useful on consumer Apple Silicon hardware with dramatically lower resident memory and/or memory bandwidth requirements.** Do not assume in advance what the solution should be. Do not force a particular architecture. Do not begin implementation before understanding the state of the art. Your job is to discover what is actually possible. --- ## 2. Core research question Investigate: > **Can an existing dense or MoE pretrained LLM be transformed post-training into an execution representation whose instantaneous working set is dramatically smaller than the full checkpoint, while preserving most or all of the original model's useful capabilities?** The most important distinction is: ```text total model size ≠ resident model size ≠ bytes read per token ≠ parameters materially required for a particular token ``` Explore whether these quantities can be decoupled. A successful system could potentially make a model much larger than available unified memory usable locally on a Mac, where additional model size primarily increases latency or storage requirements rather than making execution impossible. ### Guiding principle Do not optimize first for benchmark scores. Initially optimize for discovering whether there is a **new execution regime**. The main scientific question is: ```text How little of the original model must actually be represented, loaded, reconstructed, or evaluated to reproduce its behavior? ``` Treat this as a systems + machine-learning research problem. You are free to discover that the initial premise is wrong. **Negative experimental results are useful.** --- ## 3. Project structure (real research-project layout) The repository must follow this layout. Create the skeleton (with header-compliant placeholder files) before Phase 1 concludes. ```text localvm-research/ ├── CLAUDE.md # this charter ├── README.md # public-facing summary (written last, updated continuously) ├── CITATION.cff # citation metadata (author: Simon-Pierre Boucher) ├── LICENSE ├── pyproject.toml # Python project config (ruff, pytest, deps) ├── Makefile # top-level tasks: setup, lint, test, bench, headers │ ├── research/ # the scientific paper trail │ ├── LOG.md # dated research log (append-only) │ ├── state_of_the_art.md # Phase 2 deliverable │ ├── research_gaps.md # Phase 3 deliverable │ ├── candidate_ranking.md # Phase 4 deliverable │ ├── bibliography.md # every source, with links and access dates │ ├── novelty_check.md # Phase 11 deliverable │ └── notes/ # per-topic reading notes (one file per theme) │ ├── src/ # core library code │ ├── localvm/ │ │ ├── __init__.py │ │ ├── models/ # model loading, checkpoint parsing (safetensors/GGUF) │ │ ├── transforms/ # quantization, decomposition, encodings │ │ ├── runtime/ # paging, scheduling, caching, prediction │ │ ├── backends/ │ │ │ ├── mlx_backend/ # MLX / Metal primary path │ │ │ ├── mps_backend/ # PyTorch-MPS path │ │ │ ├── cpu_backend/ # Accelerate / NEON path │ │ │ └── cuda_optional/ # isolated, optional │ │ ├── instrumentation/ # macOS-native measurement (mach, vm_stat, fs_usage…) │ │ └── quality/ # perplexity, KL, decision-stability metrics │ └── kernels/ # custom Metal (.metal) and C++ kernels │ ├── experiments/ # micro-experiments and candidate prototypes │ ├── micro/ │ │ ├── expA_weight_concentration/ │ │ ├── expB_token_stability/ │ │ ├── expC_semantic_locality/ │ │ ├── expD_progressive_reconstruction/ │ │ ├── expE_partial_gemm/ │ │ ├── expF_error_accumulation/ │ │ ├── expG_decision_stability/ │ │ └── expH_ssd_feasibility/ │ ├── candidate_01/ │ ├── candidate_02/ │ └── candidate_03/ │ └── (each candidate contains:) │ ├── README.md │ ├── hypothesis.md │ ├── implementation/ │ ├── benchmark.py │ ├── results/ │ └── analysis.md │ ├── benchmarks/ # baselines and standardized harness │ ├── harness.py # unified benchmark runner │ ├── hardware_manifest.py # macOS hardware/software fingerprinting │ ├── baselines/ # llama.cpp, MLX, mmap, offload configs │ └── datasets/ # eval prompts: code, math, chat, FR, EN, reasoning │ ├── results/ # raw + aggregated results (JSON/CSV + plots) │ └── // │ ├── tools/ │ ├── check_headers.py # enforces §0.1 │ ├── new_experiment.py # scaffolds a compliant experiment directory │ └── report.py # generates result tables/plots │ └── docs/ # architecture docs, diagrams, final writeups ``` Every experiment directory is scaffolded by `tools/new_experiment.py`, which auto-inserts the author header and a `hypothesis.md` template containing the seven-field scientific-discipline block (§10). --- ## 4. Phase 1 — Ultra-deep web and literature research Before proposing an architecture, perform an extremely deep search of current literature, repositories, technical reports, conference papers, preprints, blog posts, implementations, issue discussions, benchmarks, and systems research. The search must be current. Search broadly and recursively. Do not stop after finding several obvious papers. Follow citations backward and forward whenever useful. Search arXiv, OpenReview, conference proceedings (MLSys, OSDI, SOSP, ASPLOS, ISCA, NeurIPS, ICML, ICLR, ACL), GitHub, Hugging Face, systems research venues, vendor engineering documentation (especially **Apple ML/Metal engineering material and MLX repos**), academic project pages, and relevant technical discussions. Prefer primary sources. For every important technique, find the actual paper and, whenever available, the implementation. Log every consulted source in `research/bibliography.md` with URL and access date. ### Areas that must be investigated At minimum investigate all of the following, while remaining open to unrelated approaches. #### 4.1 Quantization PTQ; QAT; 8-bit; 6-bit; 4-bit; 3-bit; 2-bit; 1.58-bit; ternary weights; 1-bit approaches; mixed precision; per-layer precision; per-channel precision; per-token precision; dynamic precision; progressive precision; residual quantization; recurrent residual quantization; additive quantization; vector quantization; lattice quantization; codebook methods; weight-only quantization; activation quantization; KV-cache quantization; extreme low-bit inference. Study quality degradation and actual memory-bandwidth effects **separately**. Pay particular attention to which formats have efficient Metal/MLX kernels versus which exist only on CUDA. #### 4.2 Activation sparsity Contextual sparsity; dynamic activation sparsity; FFN sparsity; neuron activation prediction; top-k neuron selection; ReLUfication; SwiGLU sparsification; structured sparsity; unstructured sparsity; N:M sparsity; activation predictors; learned sparsity routers; post-training induced sparsity. Study systems including but not limited to: DejaVu; ShadowLLM; PowerInfer; PowerInfer-2; DynamicInfer; SparQ; contextual sparsity systems. Determine exactly what is already known, and what portions assume discrete-GPU architectures that do not map to unified memory. #### 4.3 Weight sparsity and pruning SparseGPT; Wanda; magnitude pruning; structured pruning; block pruning; channel pruning; layer dropping; width pruning; depth pruning; one-shot pruning; dynamic pruning; recoverable pruning; reversible pruning; conditional pruning. Ask whether discarded weights could instead live on the Mac's NVMe SSD and be recovered only when needed. #### 4.4 Out-of-core inference CPU/GPU offload; RAM/VRAM tiering; NVMe offload; mmap; asynchronous I/O; direct I/O; page cache behavior; pinned memory; unified memory; **Apple Silicon memory behavior specifically**; GPU prefetch; tensor paging; weight streaming; SSD-to-GPU pipelines; near-storage inference; computational storage. Investigate systems such as: FlexGen; DeepSpeed inference; ZeRO-Inference; llama.cpp; MLX; PowerInfer; M2Cache; SolidAttention; Petals where relevant; distributed consumer inference systems. Measure whether the true bottleneck is capacity, bandwidth, latency, random I/O, or synchronization — **on macOS/APFS/Apple NVMe, not on Linux assumptions**. #### 4.5 Model decomposition Research whether pretrained weights can be represented as `base + residual`, `shared component + layer-specific correction`, or `low-rank component + sparse residual`. Investigate: SVD; truncated SVD; tensor decomposition; low-rank factorization; LoRA-like decomposition of existing weights; cross-layer sharing; DeltaLLM; matrix dictionaries; learned codebooks; basis decomposition; Kronecker decomposition; tensor trains; product quantization; weight clustering; block-level factorization; shared latent weight representations. Determine whether model layers contain exploitable redundancy that existing runtimes ignore. #### 4.6 Progressive and approximate computation Progressive inference; anytime neural networks; early exit; adaptive computation; dynamic depth; conditional computation; residual refinement; coarse-to-fine inference; iterative refinement; approximate matrix multiplication; error-bounded GEMM; approximate computing; lazy tensor evaluation; partial matrix multiplication; adaptive precision numerical computing. Do not limit this search to LLM research. Look at computer architecture, numerical linear algebra, databases, graphics, signal processing, scientific computing, and HPC. Potentially useful concepts may already exist outside machine learning. #### 4.7 Speculative execution Speculative decoding; self-speculative decoding; draft models; Medusa-like approaches; verification methods; exact distribution preservation; optimistic execution; rollback; branch prediction; confidence-based speculative computation. Ask whether speculation can happen **inside a transformer forward pass**, not only across future tokens. #### 4.8 Memory systems (outside AI literature) Virtual memory; demand paging; working-set theory; page replacement; TLBs; cache associativity; hardware prefetchers; branch predictors; speculative execution; NUMA; memory compression (including **macOS compressed memory**); compressed caches; tiered memory; CXL memory; object stores; database buffer pools; columnar execution; query optimizers. Ask: *What ideas from operating systems and CPU architecture have never been properly translated to neural-weight execution?* #### 4.9 Numerical error analysis Interval arithmetic; affine arithmetic; probabilistic bounds; matrix norm bounds; perturbation theory; Lipschitz bounds; error propagation; certified neural networks; robustness certification; approximate linear algebra; bounds for quantized networks. A potentially important direction: can computation terminate when additional accuracy can no longer meaningfully affect the output? Do not assume this is feasible. Test it. #### 4.10 Output-decision stability Investigate the distinction between *reproducing exact hidden states* and *producing the same useful output*. For greedy decoding, investigate whether a token decision can be certified without fully reconstructing every upstream operation. For sampling, investigate whether approximate intermediate computation can still preserve or closely approximate the original probability distribution. Explore: top-1 logit margins; top-k stability; KL divergence; total variation distance; distributional guarantees; acceptance/rejection correction; speculative verification. This is an important area but is **not prescribed as the final approach**. --- ## 5. Phase 2 — State-of-the-art map Produce `research/state_of_the_art.md` (with the §0.1 front-matter header). It must organize every relevant technique by: ```text Technique Problem addressed Model modification required? Retraining required? Memory reduction Bandwidth reduction Compute reduction Latency effect Quality degradation Hardware assumptions (explicitly: does it work on Apple Silicon / Metal?) Open-source implementation (and whether it builds on macOS arm64) Main limitation Opportunity for extension ``` Do not simply summarize papers. Identify where approaches overlap. Identify combinations that have already been tried. Identify ideas that initially seem novel but are actually already known. --- ## 6. Phase 3 — Identify genuine gaps Produce `research/research_gaps.md`. For every promising gap explain: 1. what existing systems do; 2. what they do not do; 3. why the missing capability might matter; 4. whether there is a plausible mathematical or systems reason it could work; 5. the biggest reason it might fail; 6. the smallest experiment capable of falsifying it — **runnable on a Mac**. Generate many candidate ideas. Aim for at least **20 substantially different approaches**. Do not make them superficial variants of the same idea. Example categories (non-binding): execution reordering; learned weight paging; progressive reconstruction; activation-conditioned decompression; semantic caches; low-rank hot path + residual cold path; predictive SSD reads; dynamic precision; partial GEMM; temporary model specialization; token-dependent model materialization; model-state compression; weight synthesis; cross-layer reuse; hidden-state approximation; error-controlled execution; reversible approximation; adaptive layer reconstruction. These are examples only. Do not constrain the search to them. --- ## 7. Phase 4 — Rank candidate approaches Create `research/candidate_ranking.md`. Score every idea 1–10, with written reasoning, on: ```text Novelty Technical plausibility Expected memory reduction Expected bandwidth reduction Expected quality retention Implementation complexity Hardware practicality on Apple Silicon (Metal/MLX kernel feasibility, unified memory fit) Post-training compatibility Potential research significance Risk ``` Select roughly **3–5 strongest candidates** for experiments. Prefer ideas that could fundamentally change the scaling relationship between `model size` and `required resident memory / bytes transferred per token`. --- ## 8. Phase 5 — Experimental framework Build an experimental framework **before** attempting large models. The system must make experiments reproducible. Preferred languages and stacks (Mac-first): ```text Python (MLX, PyTorch-MPS, NumPy) for research tooling C++ (AppleClang, Accelerate) where low-level control matters Metal Shading Language for custom Apple GPU kernels Swift/Objective-C shims only where mach/IOKit APIs require them CUDA only as isolated optional validation on non-Mac hardware ``` Do not prematurely optimize. Correctness and measurement come first. ### 8.1 Initial model sizes Start small. Use models in approximately this progression: ```text 0.5B–1B → 3B → 7B–8B → 14B → 32B ``` Only move to 70B+ if evidence supports it. Prefer modern open-weight architectures representative of models we ultimately want to run, with checkpoints available in safetensors and GGUF. ### 8.2 Instrumentation (macOS-native) The framework must measure, when applicable: ```text resident RAM (mach task_info) peak RAM mapped memory GPU/unified memory in use SSD reads (fs_usage / iostat) SSD bytes/token RAM bytes/token effective weight bytes/token tokens/second time-to-first-token per-token latency CPU utilization (P vs E cores) GPU utilization SSD utilization energy via powermetrics (if sudo) cache hit rate page faults (vm_stat deltas) number of parameters touched number of blocks touched effective precision used recomputation count prediction hit rate memory-pressure / compressor activity thermal state (throttling detection) ``` Also measure quality: ```text perplexity logit correlation KL divergence same greedy token rate top-k overlap task benchmark accuracy generation similarity long-context behavior coding behavior reasoning behavior ``` ### 8.3 Most important systems metric Track **BYTES READ PER GENERATED TOKEN** separately from model size. Also track **ACTIVE / TOUCHED PARAMETERS PER TOKEN** where meaningful. A system that stores a 150 GB checkpoint but reads only 5 GB per generated token may be much more interesting than one that compresses the checkpoint to 70 GB but reads all 70 GB every token. ### 8.4 Baselines At minimum compare against appropriate configurations of: ```text full precision where practical Q8 / Q6 / Q5 / Q4 / Q3 / Q2 (where practical) llama.cpp (Metal build) MLX (native) CPU offloading mmap streaming standard SSD offloading ``` Also reproduce relevant published methods when feasible. Do not claim improvement against straw-man baselines. --- ## 9. Phase 6 — Micro-experiments Before implementing a large runtime, answer fundamental questions empirically. Each lives in `experiments/micro/` with the standard scaffold. ### Experiment A — Weight contribution concentration For each transformer layer and token, measure how much of the output norm arises from subsets of weight blocks. Ask: can 10%, 20%, 40%, or 60% of blocks reproduce most of the layer output? Test across: code, math, chat, French, English, reasoning, random text, long context. ### Experiment B — Stability across consecutive tokens Measure overlap between important blocks at tokens t, t+1, t+2. Compute Jaccard similarity, transition probabilities, working-set lifetime. Determine whether weight access is predictable. ### Experiment C — Cross-prompt semantic locality Ask whether prompts belonging to the same semantic domain repeatedly rely on similar regions. If so, investigate whether domain-specific weight caches are possible. ### Experiment D — Progressive weight reconstruction Represent weights using multiple progressive approximations (2-bit only; 2+residual; 2+2 residuals; …). For each token/layer determine how rapidly hidden-state error, logit error, and the token decision converge. ### Experiment E — Partial matrix multiplication Compute only selected weight blocks. Measure quality vs blocks evaluated. Then determine whether important blocks can be predicted cheaply. Implement at least one Metal kernel variant to check that block-skipping actually saves bandwidth on Apple GPUs (not just FLOPs on paper). ### Experiment F — Error accumulation Introduce controlled approximation at individual layers. Determine which layers tolerate error, which amplify error, which recover from error. Map model sensitivity. ### Experiment G — Decision stability For an approximate forward pass, compare top logits against the exact model. Ask whether many token decisions become stable before full model precision is available. ### Experiment H — SSD feasibility (macOS-specific) Simulate realistic storage on the actual Mac's internal NVMe. Do NOT rely on theoretical bandwidth. Measure actual random and sequential reads at 4 KB, 16 KB, 64 KB, 256 KB, 1 MB, 4 MB — with and without page cache (`F_NOCACHE`), cold vs warm APFS state. Measure concurrent reads while Metal GPU compute occurs. Test whether useful prefetch overlap exists. Record SSD model and thermal state; Apple SSDs throttle. --- ## 10. Scientific discipline For every experiment explicitly write, in its `hypothesis.md` / `analysis.md`: ```text Hypothesis Falsification criterion Method Baseline Result Interpretation Next experiment ``` Avoid confirmation bias. If an idea fails, record why. **Do not silently discard failures.** ### Evidence standard Never write "this is faster / novel / better / should scale" without evidence. Use benchmark numbers. Whenever possible report: ```text mean, median, standard deviation, number of runs, hardware (chip, RAM, SSD, macOS version), model, context length, generation length, quantization, software versions (MLX, PyTorch, Metal), thermal conditions ``` --- ## 11. Phase 7 — Prototype candidate architectures For each of the strongest ideas create its own directory: ```text experiments/candidate_01/ experiments/candidate_02/ experiments/candidate_03/ ``` Each candidate must include: ```text README.md hypothesis.md implementation/ benchmark.py results/ analysis.md ``` All implementation files carry the §0.1 header and must run on the primary Mac target without CUDA. --- ## 12. Phase 8 — Automatic research loop Operate as a research agent. After every experiment: 1. analyze results; 2. determine the most informative next experiment; 3. search the literature again if results reveal something unexpected; 4. update hypotheses; 5. modify the candidate architecture; 6. rerun; 7. compare; 8. continue until evidence strongly favors or rejects the approach. Do not follow a rigid predefined implementation roadmap if experiments contradict it. ### Research log Maintain `research/LOG.md` (append-only). Every substantial action must contain: ```text date/time (local, with timezone) question experiment result interpretation decision ``` This must make the entire reasoning process auditable. --- ## 13. Phase 9 — Try to discover a new execution abstraction Do not merely seek an optimization. Seek an abstraction that could change how large local models are executed. Useful questions include: ```text Could model weights behave like virtual memory? Could only a token-specific working set be materialized? Could a model be represented as a cheap approximate core plus recoverable information? Could computation proceed before all weights are available? Could missing computation be added later only when needed? Could weights be reconstructed from shared bases? Could hidden-state trajectories predict future memory accesses? Could the system learn its own cache policy? Could model execution become demand-driven? Could the runtime optimize for decision stability rather than numerical exactness? Could a model larger than RAM become usable because RAM limits working-set size rather than total model size? Could Apple's unified memory + fast NVMe make an abstraction viable that discrete-GPU architectures cannot support? ``` These are research questions, not required design decisions. --- ## 14. Phase 10 — Build the best prototype supported by evidence Once experiments clearly favor an architecture, build a prototype runtime. A possible CLI shape could eventually be: ```bash localvm compile MODEL_PATH \ --target-memory 16GB \ --storage /Volumes/FastNVMe/localvm ``` then: ```bash localvm run COMPILED_MODEL ``` Naming and architecture should only be finalized after research. ### Compilation stage If useful, compilation may perform offline operations such as: profiling; weight analysis; quantization; factorization; clustering; reordering; block creation; index construction; activation tracing; cache-profile creation; predictor training; error-bound estimation; progressive encoding; layout optimization (aligned to APFS/Metal buffer constraints). Compilation can take substantial time. Inference must be the optimized stage. ### Hard constraint The original pretrained model must remain the source model. You may transform its representation post-training. Small auxiliary predictors, indexes, adapters, or calibration passes are allowed if justified. **Do not solve the problem by simply training a new smaller LLM to replace it.** --- ## 15. Target hardware Primary research target: ```text Apple Silicon Mac (laptop or desktop) 16–64 GB unified memory internal Apple NVMe SSD (2–8 GB/s class) Metal-capable GPU sharing memory with CPU ``` Apple Silicon is the primary platform because: ```text CPU and GPU share unified memory Metal allows custom kernels fast internal SSDs are standard large memory configurations exist (up to 128–192 GB on desktop parts) memory compression is built into the OS ``` The architecture should nevertheless remain conceptually hardware-independent; document (do not implement, unless trivial) how each mechanism would map to x86 + discrete GPU. ### Stretch target ```text model total representation: 100–250+ GB machine unified memory: 16–32 GB useful generation speed: interactive or near-interactive quality: close to original model ``` Do not assume this target is achievable. The research must establish the actual limits. --- ## 16. What counts as a breakthrough A result is scientifically interesting if at least one of the following is demonstrated: **A.** A model significantly larger than unified memory runs locally on a Mac with acceptable interactive latency. **B.** Bytes transferred per token become substantially smaller than the compressed checkpoint size. **C.** Only a small token-dependent fraction of model information is required during typical inference. **D.** A progressive or conditional execution mechanism preserves model quality while avoiding large amounts of weight loading. **E.** A new post-training representation creates a qualitatively better storage/RAM/quality tradeoff than fixed quantization. **F.** A new caching, prediction, scheduling, decomposition, or reconstruction mechanism materially changes out-of-core inference economics. ## 17. Failure criteria Be willing to conclude that an approach does not work. For example: ```text SSD bandwidth fundamentally dominates weight accesses are insufficiently predictable required working set is nearly the entire model approximation errors compound uncontrollably low-bit base models destroy routing information prediction overhead exceeds savings random I/O eliminates theoretical advantages quality degradation is unacceptable macOS I/O or Metal dispatch overhead erases theoretical wins ``` If these occur, document them in `research/LOG.md` and the relevant `analysis.md`, then pivot. --- ## 18. Deliverables Eventually produce (all with compliant headers): ```text README.md CITATION.cff research/ state_of_the_art.md research_gaps.md candidate_ranking.md novelty_check.md LOG.md bibliography.md src/ experiments/ benchmarks/ results/ docs/ tools/ ``` The final README must explain: ```text the problem what existing systems do what gap was discovered the proposed architecture why it should work experimental evidence (with numbers, on named Mac hardware) performance limitations how to reproduce (exact Mac setup instructions) future research ``` --- ## 19. Phase 11 — Novelty verification Before claiming novelty, perform a dedicated final literature search using terminology derived from the architecture actually discovered. Search for conceptual equivalents, not merely identical terminology. For example, if the architecture resembles `lazy neural execution`, `progressive weight materialization`, `conditional tensor paging`, or `activation-conditioned decompression` — search every plausible synonym. Assume an idea is **not** novel until evidence suggests otherwise. Record the process and conclusion in `research/novelty_check.md`. --- ## 20. Most important instruction Do not become attached to any particular solution suggested in this document. The purpose of this project is **not to implement a preconceived WeightVM, NeuralOS, progressive quantizer, sparse runtime, or paging engine**. Those ideas are merely clues. The actual assignment is: > **Search deeply enough, reason independently enough, and experiment aggressively enough to discover the best technically plausible way of making an existing large pretrained model usable on a Mac that normally should not be able to run it.** Start with literature. Then generate hypotheses. Then falsify them experimentally. Let evidence determine the architecture. The ideal outcome is not simply another optimization. The ideal outcome is a mechanism that changes the relationship: ```text model size → hardware required ``` into something closer to: ```text instantaneous useful working set → hardware required ``` while keeping the intelligence already present in the original pretrained model. --- *Author: Simon-Pierre Boucher — contact@spboucher.ai — All research artifacts in this repository carry this attribution.*