# CLAUDE.md

Project codename: localvm-research (provisional — final name decided by evidence, not preference) Principal investigator / Author: Simon-Pierre Boucher — contact@spboucher.ai Primary platform: Apple Silicon Mac (macOS 14+) Document status: Living research charter. Claude must treat this as the authoritative project specification.

# 0. Administrative conventions (MANDATORY — read before anything else)

# 0.1 Author header requirement

Every single source file created in this project — without exception — must begin with a standardized author header.

This applies to: Python, C++, Objective-C, Swift, Metal shaders, shell scripts, Makefiles, CMake files, configuration files that support comments, and benchmark scripts.

# Python / shell / YAML / TOML header

# =============================================================================
#  Project   : localvm-research
#  File      : <relative/path/to/file.py>
#  Purpose   : <one-line description of what this file does>
#  Author    : Simon-Pierre Boucher
#  Contact   : contact@spboucher.ai
#  Created   : <YYYY-MM-DD>
#  Modified  : <YYYY-MM-DD>
#  Platform  : macOS / Apple Silicon (arm64)
#  License   : All rights reserved (research code)
# =============================================================================

# C++ / Metal / Swift / Objective-C header

// ============================================================================
//  Project   : localvm-research
//  File      : <relative/path/to/file.cpp>
//  Purpose   : <one-line description>
//  Author    : Simon-Pierre Boucher
//  Contact   : contact@spboucher.ai
//  Created   : <YYYY-MM-DD>
//  Modified  : <YYYY-MM-DD>
//  Platform  : macOS / Apple Silicon (arm64) — Metal / Accelerate / MLX
//  License   : All rights reserved (research code)
// ============================================================================

# Markdown research documents header (front matter)

---
project: localvm-research
document: <name>
author: Simon-Pierre Boucher
contact: contact@spboucher.ai
created: <YYYY-MM-DD>
status: draft | reviewed | final
---

Rules:

1. The header must be the first content of the file (after a shebang line if one exists).
2. Modified must be updated whenever the file is substantially changed.
3. A CI-style check script (tools/check_headers.py) must be written early in the project and run before every commit; it fails if any tracked source file lacks a conforming header.
4. Generated files (e.g., results JSON) are exempt, but the generator must embed "author": "Simon-Pierre Boucher", "contact": "contact@spboucher.ai" in output metadata where format permits.

# 0.2 macOS-first design constraint

Everything in this project must be designed to run on a Mac. Specifically:

• Target machine class: Apple Silicon (M1/M2/M3/M4 family), 16–64 GB unified memory, internal NVMe SSD.
• Default compute paths: MLX, Metal (MSL kernels), Accelerate/BLAS, PyTorch with MPS backend, plain CPU (arm64 NEON).
• CUDA is allowed only as an optional, clearly isolated validation path (src/backends/cuda_optional/), never a dependency of the core runtime.
• All I/O experiments must account for macOS specifics: APFS behavior, the unified page cache, mmap semantics on macOS, F_NOCACHE/fcntl instead of Linux O_DIRECT, posix_fadvise absence, purge/vm_stat for memory pressure measurement, and Apple SSD controller characteristics.
• Instrumentation must use macOS-native sources where possible: task_info / mach APIs, vm_stat, fs_usage, powermetrics (energy, requires sudo — degrade gracefully), Instruments traces where practical, sysctl hw.* for hardware discovery.
• Build tooling: uv or pip + venv for Python; cmake + AppleClang for C++; xcrun -sdk macosx metal for Metal shader compilation. No Linux-only assumptions anywhere in the core.
• A hardware manifest (benchmarks/hardware_manifest.py) must record chip model, core counts (P/E), GPU core count, memory size, SSD model, macOS version, and software versions into every result file.

# 0.3 Repository discipline

• Git from day one. Meaningful commits. No giant "misc" commits.
• Every experiment result must be reproducible from: (commit hash) + (config file) + (seed) + (hardware manifest).
• Python: type hints, ruff for lint/format, pytest for correctness tests of numerical code.
• C++: -Wall -Wextra -Werror, sanitizers in debug builds.
• No result may be reported from an uncommitted working tree.

# 1. Mission

Your objective is to investigate, design, implement, and experimentally validate a genuinely new way to run existing pretrained open-weight large language models that normally do not fit comfortably in consumer-Mac memory on ordinary local Apple Silicon hardware.

The target problem is not:

• training a new foundation model;
• designing a smaller replacement model;
• merely applying standard quantization;
• merely using CPU offload;
• merely using SSD swap;
• merely pruning a model once and accepting permanent quality loss;
• or reproducing llama.cpp, MLX, PowerInfer, FlexGen, speculative decoding, or another existing system.

The target is broader and more ambitious:

Given an already-trained model whose normal memory and bandwidth requirements exceed the target Mac, determine whether its execution can be reorganized, transformed, compiled, approximated, decomposed, scheduled, cached, paged, predicted, refined, or otherwise executed differently so that the model remains meaningfully useful on consumer Apple Silicon hardware with dramatically lower resident memory and/or memory bandwidth requirements.

Do not assume in advance what the solution should be. Do not force a particular architecture. Do not begin implementation before understanding the state of the art. Your job is to discover what is actually possible.

# 2. Core research question

Investigate:

Can an existing dense or MoE pretrained LLM be transformed post-training into an execution representation whose instantaneous working set is dramatically smaller than the full checkpoint, while preserving most or all of the original model's useful capabilities?

The most important distinction is:

total model size
≠
resident model size
≠
bytes read per token
≠
parameters materially required for a particular token

Explore whether these quantities can be decoupled.

A successful system could potentially make a model much larger than available unified memory usable locally on a Mac, where additional model size primarily increases latency or storage requirements rather than making execution impossible.

# Guiding principle

Do not optimize first for benchmark scores. Initially optimize for discovering whether there is a new execution regime.

The main scientific question is:

How little of the original model must actually be represented,
loaded, reconstructed, or evaluated to reproduce its behavior?

Treat this as a systems + machine-learning research problem. You are free to discover that the initial premise is wrong. Negative experimental results are useful.

# 3. Project structure (real research-project layout)

The repository must follow this layout. Create the skeleton (with header-compliant placeholder files) before Phase 1 concludes.

localvm-research/
├── CLAUDE.md                     # this charter
├── README.md                     # public-facing summary (written last, updated continuously)
├── CITATION.cff                  # citation metadata (author: Simon-Pierre Boucher)
├── LICENSE
├── pyproject.toml                # Python project config (ruff, pytest, deps)
├── Makefile                      # top-level tasks: setup, lint, test, bench, headers
│
├── research/                     # the scientific paper trail
│   ├── LOG.md                    # dated research log (append-only)
│   ├── state_of_the_art.md      # Phase 2 deliverable
│   ├── research_gaps.md         # Phase 3 deliverable
│   ├── candidate_ranking.md     # Phase 4 deliverable
│   ├── bibliography.md          # every source, with links and access dates
│   ├── novelty_check.md         # Phase 11 deliverable
│   └── notes/                    # per-topic reading notes (one file per theme)
│
├── src/                          # core library code
│   ├── localvm/
│   │   ├── __init__.py
│   │   ├── models/               # model loading, checkpoint parsing (safetensors/GGUF)
│   │   ├── transforms/           # quantization, decomposition, encodings
│   │   ├── runtime/              # paging, scheduling, caching, prediction
│   │   ├── backends/
│   │   │   ├── mlx_backend/      # MLX / Metal primary path
│   │   │   ├── mps_backend/      # PyTorch-MPS path
│   │   │   ├── cpu_backend/      # Accelerate / NEON path
│   │   │   └── cuda_optional/    # isolated, optional
│   │   ├── instrumentation/      # macOS-native measurement (mach, vm_stat, fs_usage…)
│   │   └── quality/              # perplexity, KL, decision-stability metrics
│   └── kernels/                  # custom Metal (.metal) and C++ kernels
│
├── experiments/                  # micro-experiments and candidate prototypes
│   ├── micro/
│   │   ├── expA_weight_concentration/
│   │   ├── expB_token_stability/
│   │   ├── expC_semantic_locality/
│   │   ├── expD_progressive_reconstruction/
│   │   ├── expE_partial_gemm/
│   │   ├── expF_error_accumulation/
│   │   ├── expG_decision_stability/
│   │   └── expH_ssd_feasibility/
│   ├── candidate_01/
│   ├── candidate_02/
│   └── candidate_03/
│       └── (each candidate contains:)
│           ├── README.md
│           ├── hypothesis.md
│           ├── implementation/
│           ├── benchmark.py
│           ├── results/
│           └── analysis.md
│
├── benchmarks/                   # baselines and standardized harness
│   ├── harness.py                # unified benchmark runner
│   ├── hardware_manifest.py      # macOS hardware/software fingerprinting
│   ├── baselines/                # llama.cpp, MLX, mmap, offload configs
│   └── datasets/                 # eval prompts: code, math, chat, FR, EN, reasoning
│
├── results/                      # raw + aggregated results (JSON/CSV + plots)
│   └── <experiment_id>/<timestamp>/
│
├── tools/
│   ├── check_headers.py          # enforces §0.1
│   ├── new_experiment.py         # scaffolds a compliant experiment directory
│   └── report.py                 # generates result tables/plots
│
└── docs/                         # architecture docs, diagrams, final writeups

Every experiment directory is scaffolded by tools/new_experiment.py, which auto-inserts the author header and a hypothesis.md template containing the seven-field scientific-discipline block (§10).

# 4. Phase 1 — Ultra-deep web and literature research

Before proposing an architecture, perform an extremely deep search of current literature, repositories, technical reports, conference papers, preprints, blog posts, implementations, issue discussions, benchmarks, and systems research.

The search must be current. Search broadly and recursively. Do not stop after finding several obvious papers. Follow citations backward and forward whenever useful.

Search arXiv, OpenReview, conference proceedings (MLSys, OSDI, SOSP, ASPLOS, ISCA, NeurIPS, ICML, ICLR, ACL), GitHub, Hugging Face, systems research venues, vendor engineering documentation (especially Apple ML/Metal engineering material and MLX repos), academic project pages, and relevant technical discussions.

Prefer primary sources. For every important technique, find the actual paper and, whenever available, the implementation. Log every consulted source in research/bibliography.md with URL and access date.

# Areas that must be investigated

At minimum investigate all of the following, while remaining open to unrelated approaches.

# 4.1 Quantization

PTQ; QAT; 8-bit; 6-bit; 4-bit; 3-bit; 2-bit; 1.58-bit; ternary weights; 1-bit approaches; mixed precision; per-layer precision; per-channel precision; per-token precision; dynamic precision; progressive precision; residual quantization; recurrent residual quantization; additive quantization; vector quantization; lattice quantization; codebook methods; weight-only quantization; activation quantization; KV-cache quantization; extreme low-bit inference.

Study quality degradation and actual memory-bandwidth effects separately. Pay particular attention to which formats have efficient Metal/MLX kernels versus which exist only on CUDA.

# 4.2 Activation sparsity

Contextual sparsity; dynamic activation sparsity; FFN sparsity; neuron activation prediction; top-k neuron selection; ReLUfication; SwiGLU sparsification; structured sparsity; unstructured sparsity; N:M sparsity; activation predictors; learned sparsity routers; post-training induced sparsity.

Study systems including but not limited to: DejaVu; ShadowLLM; PowerInfer; PowerInfer-2; DynamicInfer; SparQ; contextual sparsity systems. Determine exactly what is already known, and what portions assume discrete-GPU architectures that do not map to unified memory.

# 4.3 Weight sparsity and pruning

SparseGPT; Wanda; magnitude pruning; structured pruning; block pruning; channel pruning; layer dropping; width pruning; depth pruning; one-shot pruning; dynamic pruning; recoverable pruning; reversible pruning; conditional pruning.

Ask whether discarded weights could instead live on the Mac's NVMe SSD and be recovered only when needed.

# 4.4 Out-of-core inference

CPU/GPU offload; RAM/VRAM tiering; NVMe offload; mmap; asynchronous I/O; direct I/O; page cache behavior; pinned memory; unified memory; Apple Silicon memory behavior specifically; GPU prefetch; tensor paging; weight streaming; SSD-to-GPU pipelines; near-storage inference; computational storage.

Investigate systems such as: FlexGen; DeepSpeed inference; ZeRO-Inference; llama.cpp; MLX; PowerInfer; M2Cache; SolidAttention; Petals where relevant; distributed consumer inference systems.

Measure whether the true bottleneck is capacity, bandwidth, latency, random I/O, or synchronization — on macOS/APFS/Apple NVMe, not on Linux assumptions.

# 4.5 Model decomposition

Research whether pretrained weights can be represented as base + residual, shared component + layer-specific correction, or low-rank component + sparse residual.

Investigate: SVD; truncated SVD; tensor decomposition; low-rank factorization; LoRA-like decomposition of existing weights; cross-layer sharing; DeltaLLM; matrix dictionaries; learned codebooks; basis decomposition; Kronecker decomposition; tensor trains; product quantization; weight clustering; block-level factorization; shared latent weight representations.

Determine whether model layers contain exploitable redundancy that existing runtimes ignore.

# 4.6 Progressive and approximate computation

Progressive inference; anytime neural networks; early exit; adaptive computation; dynamic depth; conditional computation; residual refinement; coarse-to-fine inference; iterative refinement; approximate matrix multiplication; error-bounded GEMM; approximate computing; lazy tensor evaluation; partial matrix multiplication; adaptive precision numerical computing.

Do not limit this search to LLM research. Look at computer architecture, numerical linear algebra, databases, graphics, signal processing, scientific computing, and HPC. Potentially useful concepts may already exist outside machine learning.

# 4.7 Speculative execution

Speculative decoding; self-speculative decoding; draft models; Medusa-like approaches; verification methods; exact distribution preservation; optimistic execution; rollback; branch prediction; confidence-based speculative computation.

Ask whether speculation can happen inside a transformer forward pass, not only across future tokens.

# 4.8 Memory systems (outside AI literature)

Virtual memory; demand paging; working-set theory; page replacement; TLBs; cache associativity; hardware prefetchers; branch predictors; speculative execution; NUMA; memory compression (including macOS compressed memory); compressed caches; tiered memory; CXL memory; object stores; database buffer pools; columnar execution; query optimizers.

Ask: What ideas from operating systems and CPU architecture have never been properly translated to neural-weight execution?

# 4.9 Numerical error analysis

Interval arithmetic; affine arithmetic; probabilistic bounds; matrix norm bounds; perturbation theory; Lipschitz bounds; error propagation; certified neural networks; robustness certification; approximate linear algebra; bounds for quantized networks.

A potentially important direction: can computation terminate when additional accuracy can no longer meaningfully affect the output? Do not assume this is feasible. Test it.

# 4.10 Output-decision stability

Investigate the distinction between reproducing exact hidden states and producing the same useful output.

For greedy decoding, investigate whether a token decision can be certified without fully reconstructing every upstream operation. For sampling, investigate whether approximate intermediate computation can still preserve or closely approximate the original probability distribution.

Explore: top-1 logit margins; top-k stability; KL divergence; total variation distance; distributional guarantees; acceptance/rejection correction; speculative verification.

This is an important area but is not prescribed as the final approach.

# 5. Phase 2 — State-of-the-art map

Produce research/state_of_the_art.md (with the §0.1 front-matter header).

It must organize every relevant technique by:

Technique
Problem addressed
Model modification required?
Retraining required?
Memory reduction
Bandwidth reduction
Compute reduction
Latency effect
Quality degradation
Hardware assumptions (explicitly: does it work on Apple Silicon / Metal?)
Open-source implementation (and whether it builds on macOS arm64)
Main limitation
Opportunity for extension

Do not simply summarize papers. Identify where approaches overlap. Identify combinations that have already been tried. Identify ideas that initially seem novel but are actually already known.

# 6. Phase 3 — Identify genuine gaps

Produce research/research_gaps.md.

For every promising gap explain:

1. what existing systems do;
2. what they do not do;
3. why the missing capability might matter;
4. whether there is a plausible mathematical or systems reason it could work;
5. the biggest reason it might fail;
6. the smallest experiment capable of falsifying it — runnable on a Mac.

Generate many candidate ideas. Aim for at least 20 substantially different approaches. Do not make them superficial variants of the same idea.

Example categories (non-binding): execution reordering; learned weight paging; progressive reconstruction; activation-conditioned decompression; semantic caches; low-rank hot path + residual cold path; predictive SSD reads; dynamic precision; partial GEMM; temporary model specialization; token-dependent model materialization; model-state compression; weight synthesis; cross-layer reuse; hidden-state approximation; error-controlled execution; reversible approximation; adaptive layer reconstruction.

These are examples only. Do not constrain the search to them.

# 7. Phase 4 — Rank candidate approaches

Create research/candidate_ranking.md.

Score every idea 1–10, with written reasoning, on:

Novelty
Technical plausibility
Expected memory reduction
Expected bandwidth reduction
Expected quality retention
Implementation complexity
Hardware practicality on Apple Silicon (Metal/MLX kernel feasibility, unified memory fit)
Post-training compatibility
Potential research significance
Risk

Select roughly 3–5 strongest candidates for experiments.

Prefer ideas that could fundamentally change the scaling relationship between model size and required resident memory / bytes transferred per token.

# 8. Phase 5 — Experimental framework

Build an experimental framework before attempting large models. The system must make experiments reproducible.

Preferred languages and stacks (Mac-first):

Python (MLX, PyTorch-MPS, NumPy) for research tooling
C++ (AppleClang, Accelerate) where low-level control matters
Metal Shading Language for custom Apple GPU kernels
Swift/Objective-C shims only where mach/IOKit APIs require them
CUDA only as isolated optional validation on non-Mac hardware

Do not prematurely optimize. Correctness and measurement come first.

# 8.1 Initial model sizes

Start small. Use models in approximately this progression:

0.5B–1B  →  3B  →  7B–8B  →  14B  →  32B

Only move to 70B+ if evidence supports it. Prefer modern open-weight architectures representative of models we ultimately want to run, with checkpoints available in safetensors and GGUF.

# 8.2 Instrumentation (macOS-native)

The framework must measure, when applicable:

resident RAM (mach task_info)         peak RAM
mapped memory                         GPU/unified memory in use
SSD reads (fs_usage / iostat)         SSD bytes/token
RAM bytes/token                       effective weight bytes/token
tokens/second                         time-to-first-token
per-token latency                     CPU utilization (P vs E cores)
GPU utilization                       SSD utilization
energy via powermetrics (if sudo)     cache hit rate
page faults (vm_stat deltas)          number of parameters touched
number of blocks touched              effective precision used
recomputation count                   prediction hit rate
memory-pressure / compressor activity thermal state (throttling detection)

Also measure quality:

perplexity                 logit correlation
KL divergence              same greedy token rate
top-k overlap              task benchmark accuracy
generation similarity      long-context behavior
coding behavior            reasoning behavior

# 8.3 Most important systems metric

Track BYTES READ PER GENERATED TOKEN separately from model size. Also track ACTIVE / TOUCHED PARAMETERS PER TOKEN where meaningful.

A system that stores a 150 GB checkpoint but reads only 5 GB per generated token may be much more interesting than one that compresses the checkpoint to 70 GB but reads all 70 GB every token.

# 8.4 Baselines

At minimum compare against appropriate configurations of:

full precision where practical
Q8 / Q6 / Q5 / Q4 / Q3 / Q2 (where practical)
llama.cpp (Metal build)
MLX (native)
CPU offloading
mmap streaming
standard SSD offloading

Also reproduce relevant published methods when feasible. Do not claim improvement against straw-man baselines.

# 9. Phase 6 — Micro-experiments

Before implementing a large runtime, answer fundamental questions empirically. Each lives in experiments/micro/ with the standard scaffold.

# Experiment A — Weight contribution concentration

For each transformer layer and token, measure how much of the output norm arises from subsets of weight blocks. Ask: can 10%, 20%, 40%, or 60% of blocks reproduce most of the layer output? Test across: code, math, chat, French, English, reasoning, random text, long context.

# Experiment B — Stability across consecutive tokens

Measure overlap between important blocks at tokens t, t+1, t+2. Compute Jaccard similarity, transition probabilities, working-set lifetime. Determine whether weight access is predictable.

# Experiment C — Cross-prompt semantic locality

Ask whether prompts belonging to the same semantic domain repeatedly rely on similar regions. If so, investigate whether domain-specific weight caches are possible.

# Experiment D — Progressive weight reconstruction

Represent weights using multiple progressive approximations (2-bit only; 2+residual; 2+2 residuals; …). For each token/layer determine how rapidly hidden-state error, logit error, and the token decision converge.

# Experiment E — Partial matrix multiplication

Compute only selected weight blocks. Measure quality vs blocks evaluated. Then determine whether important blocks can be predicted cheaply. Implement at least one Metal kernel variant to check that block-skipping actually saves bandwidth on Apple GPUs (not just FLOPs on paper).

# Experiment F — Error accumulation

Introduce controlled approximation at individual layers. Determine which layers tolerate error, which amplify error, which recover from error. Map model sensitivity.

# Experiment G — Decision stability

For an approximate forward pass, compare top logits against the exact model. Ask whether many token decisions become stable before full model precision is available.

# Experiment H — SSD feasibility (macOS-specific)

Simulate realistic storage on the actual Mac's internal NVMe. Do NOT rely on theoretical bandwidth. Measure actual random and sequential reads at 4 KB, 16 KB, 64 KB, 256 KB, 1 MB, 4 MB — with and without page cache (F_NOCACHE), cold vs warm APFS state. Measure concurrent reads while Metal GPU compute occurs. Test whether useful prefetch overlap exists. Record SSD model and thermal state; Apple SSDs throttle.

# 10. Scientific discipline

For every experiment explicitly write, in its hypothesis.md / analysis.md:

Hypothesis
Falsification criterion
Method
Baseline
Result
Interpretation
Next experiment

Avoid confirmation bias. If an idea fails, record why. Do not silently discard failures.

# Evidence standard

Never write "this is faster / novel / better / should scale" without evidence. Use benchmark numbers. Whenever possible report:

mean, median, standard deviation, number of runs,
hardware (chip, RAM, SSD, macOS version), model, context length,
generation length, quantization, software versions (MLX, PyTorch, Metal),
thermal conditions

# 11. Phase 7 — Prototype candidate architectures

For each of the strongest ideas create its own directory:

experiments/candidate_01/
experiments/candidate_02/
experiments/candidate_03/

Each candidate must include:

README.md          hypothesis.md
implementation/    benchmark.py
results/           analysis.md

All implementation files carry the §0.1 header and must run on the primary Mac target without CUDA.

# 12. Phase 8 — Automatic research loop

Operate as a research agent. After every experiment:

1. analyze results;
2. determine the most informative next experiment;
3. search the literature again if results reveal something unexpected;
4. update hypotheses;
5. modify the candidate architecture;
6. rerun;
7. compare;
8. continue until evidence strongly favors or rejects the approach.

Do not follow a rigid predefined implementation roadmap if experiments contradict it.

# Research log

Maintain research/LOG.md (append-only). Every substantial action must contain:

date/time (local, with timezone)
question
experiment
result
interpretation
decision

This must make the entire reasoning process auditable.

# 13. Phase 9 — Try to discover a new execution abstraction

Do not merely seek an optimization. Seek an abstraction that could change how large local models are executed. Useful questions include:

Could model weights behave like virtual memory?
Could only a token-specific working set be materialized?
Could a model be represented as a cheap approximate core plus recoverable information?
Could computation proceed before all weights are available?
Could missing computation be added later only when needed?
Could weights be reconstructed from shared bases?
Could hidden-state trajectories predict future memory accesses?
Could the system learn its own cache policy?
Could model execution become demand-driven?
Could the runtime optimize for decision stability rather than numerical exactness?
Could a model larger than RAM become usable because RAM limits working-set size
rather than total model size?
Could Apple's unified memory + fast NVMe make an abstraction viable that
discrete-GPU architectures cannot support?

These are research questions, not required design decisions.

# 14. Phase 10 — Build the best prototype supported by evidence

Once experiments clearly favor an architecture, build a prototype runtime. A possible CLI shape could eventually be:

localvm compile MODEL_PATH \
    --target-memory 16GB \
    --storage /Volumes/FastNVMe/localvm

then:

localvm run COMPILED_MODEL

Naming and architecture should only be finalized after research.

# Compilation stage

If useful, compilation may perform offline operations such as: profiling; weight analysis; quantization; factorization; clustering; reordering; block creation; index construction; activation tracing; cache-profile creation; predictor training; error-bound estimation; progressive encoding; layout optimization (aligned to APFS/Metal buffer constraints).

Compilation can take substantial time. Inference must be the optimized stage.

# Hard constraint

The original pretrained model must remain the source model. You may transform its representation post-training. Small auxiliary predictors, indexes, adapters, or calibration passes are allowed if justified. Do not solve the problem by simply training a new smaller LLM to replace it.

# 15. Target hardware

Primary research target:

Apple Silicon Mac (laptop or desktop)
16–64 GB unified memory
internal Apple NVMe SSD (2–8 GB/s class)
Metal-capable GPU sharing memory with CPU

Apple Silicon is the primary platform because:

CPU and GPU share unified memory
Metal allows custom kernels
fast internal SSDs are standard
large memory configurations exist (up to 128–192 GB on desktop parts)
memory compression is built into the OS

The architecture should nevertheless remain conceptually hardware-independent; document (do not implement, unless trivial) how each mechanism would map to x86 + discrete GPU.

# Stretch target

model total representation: 100–250+ GB
machine unified memory:     16–32 GB
useful generation speed:    interactive or near-interactive
quality:                    close to original model

Do not assume this target is achievable. The research must establish the actual limits.

# 16. What counts as a breakthrough

A result is scientifically interesting if at least one of the following is demonstrated:

A. A model significantly larger than unified memory runs locally on a Mac with acceptable interactive latency. B. Bytes transferred per token become substantially smaller than the compressed checkpoint size. C. Only a small token-dependent fraction of model information is required during typical inference. D. A progressive or conditional execution mechanism preserves model quality while avoiding large amounts of weight loading. E. A new post-training representation creates a qualitatively better storage/RAM/quality tradeoff than fixed quantization. F. A new caching, prediction, scheduling, decomposition, or reconstruction mechanism materially changes out-of-core inference economics.

# 17. Failure criteria

Be willing to conclude that an approach does not work. For example:

SSD bandwidth fundamentally dominates
weight accesses are insufficiently predictable
required working set is nearly the entire model
approximation errors compound uncontrollably
low-bit base models destroy routing information
prediction overhead exceeds savings
random I/O eliminates theoretical advantages
quality degradation is unacceptable
macOS I/O or Metal dispatch overhead erases theoretical wins

If these occur, document them in research/LOG.md and the relevant analysis.md, then pivot.

# 18. Deliverables

Eventually produce (all with compliant headers):

README.md
CITATION.cff
research/
    state_of_the_art.md
    research_gaps.md
    candidate_ranking.md
    novelty_check.md
    LOG.md
    bibliography.md
src/
experiments/
benchmarks/
results/
docs/
tools/

The final README must explain:

the problem
what existing systems do
what gap was discovered
the proposed architecture
why it should work
experimental evidence (with numbers, on named Mac hardware)
performance
limitations
how to reproduce (exact Mac setup instructions)
future research

# 19. Phase 11 — Novelty verification

Before claiming novelty, perform a dedicated final literature search using terminology derived from the architecture actually discovered. Search for conceptual equivalents, not merely identical terminology.

For example, if the architecture resembles lazy neural execution, progressive weight materialization, conditional tensor paging, or activation-conditioned decompression — search every plausible synonym.

Assume an idea is not novel until evidence suggests otherwise. Record the process and conclusion in research/novelty_check.md.

# 20. Most important instruction

Do not become attached to any particular solution suggested in this document.

The purpose of this project is not to implement a preconceived WeightVM, NeuralOS, progressive quantizer, sparse runtime, or paging engine. Those ideas are merely clues.

The actual assignment is:

Search deeply enough, reason independently enough, and experiment aggressively enough to discover the best technically plausible way of making an existing large pretrained model usable on a Mac that normally should not be able to run it.

Start with literature. Then generate hypotheses. Then falsify them experimentally. Let evidence determine the architecture.

The ideal outcome is not simply another optimization. The ideal outcome is a mechanism that changes the relationship:

model size → hardware required

into something closer to:

instantaneous useful working set → hardware required

while keeping the intelligence already present in the original pretrained model.

Author: Simon-Pierre Boucher — contact@spboucher.ai — All research artifacts in this repository carry this attribution.