spb/modelmap Public License

1# CLAUDE.md23**Project codename:** `modelmap` — Internal Cartography of Local Large Language Models4**Public site:** https://modelmap.io (the living research atlas & documentation site)5**Principal investigator / Author:** Simon-Pierre Boucher — <contact@spboucher.ai>6**Primary platform:** Apple Silicon Mac (macOS 14+)7**Sister project:** `localvm-research` (out-of-core execution) — findings must cross-pollinate8**Document status:** Living research charter. Claude must treat this as the authoritative project specification.910---1112## 0. Administrative conventions (MANDATORY — read before anything else)1314### 0.1 Author header requirement1516**Every single source file created in this project — without exception — must begin with a standardized author header.**1718This applies to: Python, C++, Swift, Objective-C, Metal shaders, JavaScript/TypeScript (for the modelmap.io site), shell scripts, Makefiles, CMake files, and any configuration file that supports comments.1920#### Python / shell / YAML / TOML header2122```python23# =============================================================================24#  Project   : modelmap25#  File      : <relative/path/to/file.py>26#  Purpose   : <one-line description of what this file does>27#  Author    : Simon-Pierre Boucher28#  Contact   : contact@spboucher.ai29#  Website   : https://modelmap.io30#  Created   : <YYYY-MM-DD>31#  Modified  : <YYYY-MM-DD>32#  Platform  : macOS / Apple Silicon (arm64)33#  License   : All rights reserved (research code)34# =============================================================================35```3637#### C++ / Metal / Swift / JS / TS header3839```cpp40// ============================================================================41//  Project   : modelmap42//  File      : <relative/path/to/file>43//  Purpose   : <one-line description>44//  Author    : Simon-Pierre Boucher45//  Contact   : contact@spboucher.ai46//  Website   : https://modelmap.io47//  Created   : <YYYY-MM-DD>48//  Modified  : <YYYY-MM-DD>49//  Platform  : macOS / Apple Silicon (arm64) — MLX / Metal / MPS50//  License   : All rights reserved (research code)51// ============================================================================52```5354#### Markdown research documents (front matter)5556```markdown57---58project: modelmap59document: <name>60author: Simon-Pierre Boucher61contact: contact@spboucher.ai62website: https://modelmap.io63created: <YYYY-MM-DD>64status: draft | reviewed | final65---66```6768Rules:69701. The header must be the **first content** of the file (after a shebang if present).712. `Modified` must be updated on every substantial change.723. Write `tools/check_headers.py` early; it must fail if any tracked source file lacks a conforming header, and it runs before every commit.734. Generated artifacts (maps, JSON atlases, plots) must embed `"author": "Simon-Pierre Boucher", "contact": "contact@spboucher.ai", "website": "https://modelmap.io"` in their metadata wherever the format permits.745. Every published map on modelmap.io must carry a visible attribution footer.7576### 0.2 macOS-first design constraint7778**Everything must be designed to run on a Mac.** Specifically:7980* Target machine: Apple Silicon (M1–M4 family), 16–64 GB unified memory, internal NVMe SSD.81* Default compute paths: **MLX**, **PyTorch with MPS backend**, **NumPy/Accelerate**, custom **Metal** kernels where hooks/probes need speed. CUDA is allowed only as an isolated optional validation path (`src/backends/cuda_optional/`), never a core dependency.82* Activation capture at scale generates large tensors: design the capture pipeline for **unified memory limits** (streaming to disk, memory-mapped activation stores, chunked capture) — never assume activations fit in RAM.83* Storage of maps/activations must respect APFS behavior and Apple NVMe characteristics; use efficient columnar/array formats (safetensors, zarr, parquet, HDF5) benchmarked **on macOS**, not assumed from Linux numbers.84* Instrumentation must use macOS-native sources: mach `task_info`, `vm_stat`, `fs_usage`, `powermetrics` (graceful degradation without sudo), `sysctl hw.*`.85* Build tooling: `uv`/`pip` + venv for Python; `cmake` + AppleClang for C++; `xcrun -sdk macosx metal` for shaders; standard Node toolchain for the modelmap.io site (static-site friendly — the site must build and preview locally on the Mac).86* A hardware manifest (`benchmarks/hardware_manifest.py`) records chip, P/E core counts, GPU cores, RAM, SSD model, macOS version, and all software versions into every result and every published map.8788### 0.3 Repository discipline8990* Git from day one; meaningful commits; no result reported from an uncommitted tree.91* Every map must be reproducible from: commit hash + config file + model hash + seed + hardware manifest.92* Python: type hints, `ruff`, `pytest` (numerical probes get correctness tests against tiny reference models).93* Datasets of prompts used for mapping are versioned artifacts with checksums.94* Large binary artifacts (activation dumps) live outside git (local artifact store with an index file committed to git).9596---9798## 1. Mission99100Your objective is to investigate, design, implement, and experimentally validate a systematic methodology — and the tooling behind it — to **discover, measure, and map the internal structure of pretrained open-weight LLMs running locally on consumer Apple Silicon hardware**, and to publish the resulting maps as a rigorous, reproducible public atlas at **modelmap.io**.101102The target problem is **not**:103104* training new models;105* fine-tuning models to change their behavior;106* reproducing an existing interpretability library (TransformerLens, NNsight, SAELens, Captum) without adding anything;107* producing pretty but unfalsifiable visualizations;108* benchmarking model *outputs* (that is leaderboard work, not cartography);109* speculative "the model thinks X" storytelling without measurement.110111The target is broader and more rigorous:112113> **Given an already-trained open-weight model, determine what can actually be known, measured, localized, and mapped about its internal organization — where knowledge lives, how computation is distributed, which structures are stable across inputs, layers, scales, and model families — using only local, consumer-grade Mac hardware, and turn that knowledge into reproducible, comparable, publishable maps.**114115Do not assume in advance which mapping techniques are informative.116Do not force a particular interpretability framework.117Do not begin implementation before understanding the state of the art.118Your job is to discover what can genuinely be mapped — and what cannot.119120---121122## 2. Core research questions123124Investigate:125126> **Q1 — Localization.** Where inside a pretrained LLM do specific capabilities, knowledge domains, languages, and behaviors reside? Are they localized (layers, heads, neurons, weight blocks, feature directions) or diffuse?127128> **Q2 — Structure.** What stable internal structures exist across inputs — circuits, feature directions, attention patterns, activation manifolds, weight-space geometry — and at what granularity are they real rather than artifacts of the probing method?129130> **Q3 — Comparability.** Can internal maps be compared across model sizes, checkpoints, quantization levels, and model families? Is there a common "coordinate system" for model internals?131132> **Q4 — Cost.** Which mapping techniques are feasible on a 16–64 GB Mac, at which model sizes, and what is the accuracy/cost frontier of local interpretability?133134> **Q5 — Utility.** Do the maps predict anything useful — quantization sensitivity, pruning tolerance, working-set behavior (feeding `localvm-research`), failure modes, editing targets?135136The most important distinction to keep sharp throughout:137138```text139what a probing method reports140≠141what the model actually computes142≠143what is stable across methods, seeds, and datasets144≠145what is causally verified by intervention146```147148A claim only enters the atlas at the confidence level its evidence supports. Correlational maps and causally-verified maps must never be visually or textually conflated.149150### Guiding principle151152Do not optimize first for spectacular visuals. Initially optimize for discovering **which measurements are real** — reproducible, method-robust, causally grounded.153154Negative results ("technique X produces maps that do not replicate across seeds") are first-class publishable findings on modelmap.io.155156---157158## 3. Project structure (real research-project layout)159160Create this skeleton (with header-compliant placeholder files) before Phase 1 concludes:161162```text163modelmap/164├── CLAUDE.md                      # this charter165├── README.md                      # public summary (written last, updated continuously)166├── CITATION.cff                   # author: Simon-Pierre Boucher167├── LICENSE168├── pyproject.toml169├── Makefile                       # setup, lint, test, capture, map, site, headers170│171├── research/                      # the scientific paper trail172│   ├── LOG.md                     # dated, append-only research log173│   ├── state_of_the_art.md       # Phase 2 deliverable174│   ├── research_gaps.md          # Phase 3 deliverable175│   ├── candidate_ranking.md      # Phase 4 deliverable176│   ├── methodology.md            # the formal mapping methodology (Phase 9)177│   ├── novelty_check.md          # Phase 11 deliverable178│   ├── bibliography.md           # every source, links + access dates179│   └── notes/                     # per-topic reading notes180│181├── src/182│   └── modelmap/183│       ├── __init__.py184│       ├── models/                # loading (safetensors/GGUF), architecture introspection185│       ├── capture/               # activation hooks, streaming capture, mmap stores186│       ├── probes/                # linear probes, logit lens, tuned lens, causal probes187│       ├── interventions/         # ablation, patching, steering, weight zeroing188│       ├── features/              # SAE training/inference, dictionary methods189│       ├── geometry/              # weight-space analysis: SVD, spectra, similarity (CKA…)190│       ├── graphs/                # circuit extraction, attribution graphs191│       ├── atlas/                 # map schema, serialization, versioning, provenance192│       ├── viz/                   # figure + interactive-map generation193│       ├── backends/              # mlx_backend / mps_backend / cpu_backend / cuda_optional194│       ├── instrumentation/       # macOS-native cost measurement195│       └── stats/                 # replication tests, bootstrap CIs, multiple-comparison control196│197├── experiments/198│   ├── micro/199│   │   ├── expA_probe_reliability/200│   │   ├── expB_localization_vs_diffusion/201│   │   ├── expC_causal_verification/202│   │   ├── expD_cross_input_stability/203│   │   ├── expE_weight_geometry/204│   │   ├── expF_quantization_map_drift/205│   │   ├── expG_cross_model_alignment/206│   │   └── expH_capture_cost_frontier/207│   ├── candidate_01/208│   ├── candidate_02/209│   └── candidate_03/210│       └── (each: README.md, hypothesis.md, implementation/,211│                benchmark.py, results/, analysis.md)212│213├── atlas/                         # the maps themselves (versioned data products)214│   └── <model_id>/<map_type>/<version>/215│       ├── map.json / map.zarr216│       ├── provenance.json        # commit, config, model hash, hardware, dates217│       └── confidence.md          # evidence level: correlational | replicated | causal218│219├── site/                          # modelmap.io source (static-site, builds locally on Mac)220│   ├── content/                   # one page per model, per map type, per finding221│   ├── components/                # interactive map viewers222│   └── data/                      # published atlas exports223│224├── benchmarks/225│   ├── harness.py226│   ├── hardware_manifest.py227│   └── promptsets/                # versioned mapping corpora: code, math, FR, EN, facts…228│229├── results/                       # raw + aggregated results230│   └── <experiment_id>/<timestamp>/231│232├── tools/233│   ├── check_headers.py234│   ├── new_experiment.py          # scaffolds compliant experiment dirs235│   ├── new_map.py                 # scaffolds a compliant atlas entry with provenance236│   └── publish.py                 # atlas → site/data export with attribution237│238└── docs/239```240241Every experiment directory is scaffolded by `tools/new_experiment.py` with the seven-field scientific block (§10). Every atlas entry is scaffolded by `tools/new_map.py` and **cannot be published without a completed `provenance.json` and `confidence.md`**.242243---244245## 4. Phase 1 — Ultra-deep literature research246247Before proposing a methodology, perform an extremely deep search of current literature, repositories, technical reports, papers, preprints, blog posts, implementations, and discussions.248249Search arXiv, OpenReview, conference proceedings (NeurIPS, ICML, ICLR, ACL, EMNLP), the mechanistic-interpretability community corpus (Anthropic interpretability publications, Transformer Circuits thread, Neel Nanda's work, EleutherAI, LessWrong/Alignment Forum technical posts), GitHub (TransformerLens, NNsight, SAELens, Penzai, Captum, baukit), Hugging Face, and Apple/MLX engineering material.250251Prefer primary sources. For every important technique find the paper **and** the implementation, and note whether it runs on Apple Silicon. Log everything in `research/bibliography.md` with URLs and access dates. Follow citations backward and forward.252253### Areas that must be investigated254255#### 4.1 Observational probing256Linear probes; logit lens; tuned lens; early decoding; representation reading; concept erasure (LEACE-style); probing classifier pitfalls and controls; selectivity/controls literature (what probes overfit to).257258#### 4.2 Causal / interventional methods259Activation patching; path patching; attribution patching; causal tracing (ROME-style locating); ablations (zero, mean, resample); causal scrubbing; interchange interventions; steering vectors; representation engineering; distributed alignment search.260261#### 4.3 Feature decomposition262Sparse autoencoders (SAEs) and their variants (gated, top-k, jumpReLU, crosscoders, transcoders); dictionary learning; superposition theory; polysemanticity; feature splitting; SAE evaluation problems (reconstruction vs interpretability vs faithfulness); automated feature labeling and its failure modes.263264#### 4.4 Circuits and computational graphs265Induction heads; IOI-style circuit analyses; attention head taxonomies; MLP key–value memory view; attribution graphs; automated circuit discovery (ACDC and successors); edge/node pruning of computational graphs; faithfulness metrics for circuits.266267#### 4.5 Weight-space cartography (no forward pass required)268SVD spectra of weight matrices; effective rank per layer; weight statistics across depth; outlier channels/dimensions; cross-layer similarity (CKA, CCA, Procrustes); model stitching; permutation alignment / git re-basin; mode connectivity; delta analysis between checkpoints and between base/instruct variants; MoE router and expert structure.269270#### 4.6 Knowledge localization and editing271ROME, MEMIT and successors; where facts are stored vs where they are retrieved; the localization-vs-editing critique literature (does editing success actually imply localization?); knowledge neurons; multilingual knowledge sharing.272273#### 4.7 Cross-model and cross-scale comparison274Representation similarity across scales and families; universality of features/circuits; platonic representation hypothesis debates; tokenizer effects on comparability; quantization effects on internal representations (largely unmapped — note carefully).275276#### 4.8 Statistics and epistemology of interpretability277Illusions in interpretability results; seed sensitivity; dataset sensitivity of probes; multiple-comparison problems when scanning thousands of neurons/heads; pre-registration analogues; faithfulness vs plausibility; benchmarking interpretability methods (e.g., finding planted circuits in known models).278279#### 4.9 Systems side (local mapping at scale)280Activation caching formats and costs; hook overhead in MLX vs PyTorch-MPS; streaming SAE training on limited memory; disk layouts for random access into activation stores; what interpretability workloads exist for GGUF/quantized runtimes (llama.cpp introspection); GPU capture on Metal.281282#### 4.10 Visualization and atlas design283Prior art on model atlases (Neuroscope, Neuronpedia, OpenAI microscope-style efforts, feature browsers); what made them useful or misleading; provenance and versioning practices for data products; how to encode uncertainty visually.284285---286287## 5. Phase 2 — State-of-the-art map288289Produce `research/state_of_the_art.md`. Organize every relevant technique by:290291```text292Technique293What it claims to reveal294Observational or causal?295Model access required (weights / activations / gradients)296Compute + memory cost class297Feasible on 16–64 GB Apple Silicon? (measured or estimated, at which model sizes)298Known failure modes / illusions299Reproducibility record300Open-source implementation (and does it build on macOS arm64 / MPS / MLX?)301Main limitation302Opportunity for extension303```304305Do not simply summarize papers. Identify overlaps, already-tried combinations, and ideas that seem novel but are known. Explicitly flag the epistemic status of each technique (established / contested / debunked-in-part).306307---308309## 6. Phase 3 — Identify genuine gaps310311Produce `research/research_gaps.md`. For every promising gap explain:3123131. what existing work does;3142. what it does not do;3153. why the missing capability might matter;3164. whether there is a plausible mathematical or empirical reason it could work;3175. the biggest reason it might fail;3186. the smallest experiment capable of falsifying it — **runnable on a Mac**.319320Generate at least **20 substantially different candidate directions**. Do not make them superficial variants.321322Example categories (non-binding): quantization-aware internal maps; weight-only cheap cartography as a pre-screen for expensive activation methods; cross-model coordinate systems; maps that predict systems properties (working sets, pruning tolerance — bridge to `localvm`); method-agreement scoring (map = intersection of techniques); longitudinal maps across a model family; multilingual localization atlases; local-hardware-feasible SAE recipes; causal-verification pipelines that scale down; uncertainty-first atlas schemas.323324These are examples only. Do not constrain the search to them.325326---327328## 7. Phase 4 — Rank candidate approaches329330Create `research/candidate_ranking.md`. Score every idea 1–10 with written reasoning on:331332```text333Scientific value (what new knowledge does the map contain?)334Epistemic soundness (can claims be causally verified?)335Novelty336Feasibility on Apple Silicon (memory, time, kernel availability)337Reproducibility potential338Cross-model generality339Public atlas value (is it worth publishing on modelmap.io?)340Implementation complexity341Synergy with localvm-research342Risk343```344345Select roughly **3–5 strongest candidates** for prototyping.346347Prefer directions that produce **comparable, versionable, falsifiable map artifacts** rather than one-off analyses.348349---350351## 8. Phase 5 — Experimental framework352353Build the framework **before** mapping large models. Correctness and measurement first; optimization later.354355Preferred stack (Mac-first):356357```text358Python (MLX, PyTorch-MPS, NumPy, safetensors, zarr) for research tooling359C++ / Metal where hook or SAE throughput demands it360TypeScript + static-site generator for modelmap.io (local build/preview on the Mac)361CUDA only as isolated optional validation362```363364### 8.1 Model progression365366```text3670.5B–1B  →  3B  →  7B–8B  →  14B  →  32B368```369370Move larger only when evidence and the cost frontier (Experiment H) support it. Prefer modern open-weight families with multiple sizes (to enable cross-scale maps) and both base + instruct checkpoints (to enable delta maps).371372### 8.2 Instrumentation373374Measure, for every mapping run:375376```text377resident/peak RAM              activation bytes captured378disk bytes written/read        capture overhead vs plain inference379wall-clock per mapped unit     GPU/CPU utilization (P vs E cores)380hook dispatch overhead         SAE training throughput381thermal state / throttling     energy via powermetrics when available382```383384And for every scientific result:385386```text387replication across ≥3 seeds        replication across ≥2 prompt sets388bootstrap confidence intervals      effect sizes, not just p-values389method agreement (≥2 techniques)    causal verification status390multiple-comparison correction when scanning many units391```392393### 8.3 Most important scientific metric394395Track, for every map:396397```text398REPLICATION RATE (does the map reproduce under resampling of seeds/data?)399CAUSAL CONFIRMATION RATE (what fraction of localized claims survive intervention?)400```401402A beautiful map that does not replicate is worth less than a boring one that does.403404### 8.4 Baselines and controls405406Every mapping technique must be run with controls: shuffled labels for probes; random-direction baselines for steering; randomly-initialized-model baselines where meaningful; resample ablations vs zero ablations. Never claim structure without showing the null.407408---409410## 9. Phase 6 — Micro-experiments411412Each lives in `experiments/micro/` with the standard scaffold.413414### Experiment A — Probe reliability415For a small model, train linear probes for a set of properties across all layers, with shuffled-label controls, ≥5 seeds, ≥2 datasets. Quantify how much probe accuracy varies with seed/data. Establish the noise floor every later map must exceed.416417### Experiment B — Localization vs diffusion418For selected capabilities (a language, a knowledge domain, arithmetic, code syntax), measure how concentrated the supporting signal is across layers/heads/neurons/weight blocks. Ask: does removing the top-k localized units remove the capability, or does it survive (diffusion)? Test across code, math, chat, French, English, factual recall, reasoning.419420### Experiment C — Causal verification pipeline421Take the top findings from A/B and verify them with activation patching and ablation. Measure the correlational→causal survival rate. This number calibrates all future confidence labels in the atlas.422423### Experiment D — Cross-input stability424For fixed internal structures (heads, features, directions), measure stability across prompts, domains, context lengths, and sampling temperature. Compute overlap/Jaccard of "important units" across inputs. Determine which map types are input-conditional vs input-invariant.425426### Experiment E — Weight-space geometry (zero-forward-pass maps)427Compute per-layer SVD spectra, effective ranks, outlier dimensions, and cross-layer CKA purely from weights. Test whether these cheap maps predict anything measured in A–D. If yes, weight-only cartography becomes a fast pre-screen for any model.428429### Experiment F — Quantization map drift430Map the same model at FP16, Q8, Q4, Q2 (where practical). Measure how much probes, feature directions, and localized circuits drift under quantization. This is under-explored territory and directly relevant to local models, which are almost always quantized.431432### Experiment G — Cross-model alignment433For two sizes of the same family, attempt to align internal maps (representation similarity, feature matching, permutation alignment). Determine whether a shared coordinate system is achievable and at what granularity.434435### Experiment H — Capture cost frontier (macOS-specific)436Measure the real cost of activation capture on the Mac: hook overhead in MLX vs PyTorch-MPS, streaming write throughput to the internal NVMe, storage formats (safetensors vs zarr vs raw mmap), maximum feasible capture per model size within 16/32/64 GB. Produce a published "what can you map on which Mac" table — itself a modelmap.io artifact.437438---439440## 10. Scientific discipline441442For every experiment write, in `hypothesis.md` / `analysis.md`:443444```text445Hypothesis446Falsification criterion447Method (including controls)448Baseline / null449Result (with effect sizes and CIs)450Interpretation (with explicit confidence level)451Next experiment452```453454Avoid confirmation bias. If a technique produces illusory structure, record it — that is a finding. **Do not silently discard failures.**455456### Evidence standard457458Never write "this neuron does X", "this layer stores Y", "feature Z represents W" without stating: the method, the controls, the replication status, and whether the claim is correlational or causally verified. Report: number of seeds, prompt sets, effect sizes, CIs, hardware, model+quantization, software versions.459460### Confidence taxonomy (used everywhere, including on the site)461462```text463Level 0 — anecdotal        (single run, no controls; never published alone)464Level 1 — correlational    (controlled, replicated ≥3 seeds, ≥2 datasets)465Level 2 — method-robust    (Level 1 + agreement across ≥2 independent techniques)466Level 3 — causal           (Level 2 + intervention confirms the claim)467```468469Every atlas entry carries its level in `confidence.md` and visibly on modelmap.io.470471---472473## 11. Phase 7 — Prototype candidate methodologies474475For each of the strongest candidates create `experiments/candidate_0N/` with the standard contents. All implementations must run on the primary Mac target without CUDA. Each candidate's `analysis.md` must end with an explicit recommendation: promote to atlas pipeline, iterate, or reject (with reasons).476477---478479## 12. Phase 8 — Automatic research loop480481Operate as a research agent. After every experiment: analyze; pick the most informative next experiment; re-search the literature when results surprise; update hypotheses; modify the methodology; rerun; compare; continue until evidence strongly favors or rejects the approach. Do not follow a rigid roadmap if experiments contradict it.482483Maintain `research/LOG.md` (append-only): date/time with timezone, question, experiment, result, interpretation, decision. The entire reasoning process must be auditable.484485---486487## 13. Phase 9 — Formalize the mapping methodology488489Once micro-experiments and candidates converge, write `research/methodology.md`: the formal, reusable pipeline that turns *(model checkpoint + prompt corpora + budget)* into *(versioned atlas entries with confidence levels)*.490491It must specify: capture protocol; probe/feature/circuit procedures actually retained; mandatory controls; replication requirements; causal-verification requirements per confidence level; cost model per Mac tier; map schema; provenance schema; publication checklist.492493The methodology is itself a primary deliverable — it should let any researcher with a Mac reproduce a map.494495---496497## 14. Phase 10 — Build the atlas pipeline and modelmap.io498499Build the production pipeline supported by evidence. A possible CLI shape (finalize only after research):500501```bash502modelmap capture MODEL_PATH --promptset benchmarks/promptsets/core --out atlas/503modelmap analyze atlas/<model_id> --maps probes,features,geometry,circuits504modelmap verify  atlas/<model_id> --level causal505modelmap publish atlas/<model_id> --site site/506```507508### The site (modelmap.io)509510* Static-site architecture, built and previewed locally on the Mac; every page generated from atlas data, never hand-edited numbers.511* One page per model; per map type; interactive viewers (layer/head/feature browsers) built from `atlas/` exports.512* Every visualization displays: confidence level, provenance (commit, model hash, date, hardware), and how to reproduce.513* A methodology section publishing `research/methodology.md`.514* A negative-results section. Publishing what does **not** replicate is part of the mission.515* Attribution footer on every page: *Simon-Pierre Boucher — contact@spboucher.ai*.516517### Hard constraints518519* Only open-weight models are mapped; the original checkpoint is always the object of study.520* No claim appears on modelmap.io above its evidence level.521* Nothing is published that cannot be regenerated from committed code + versioned data.522523---524525## 15. Target hardware526527Primary research target:528529```text530Apple Silicon Mac (laptop or desktop)53116–64 GB unified memory532internal Apple NVMe SSD533Metal-capable GPU sharing memory with CPU534```535536Apple Silicon matters here because: unified memory lets capture pipelines avoid device copies; MLX enables custom, low-overhead hooks; fast internal SSDs make large activation stores workable; and — crucially — **local interpretability on consumer hardware is itself an under-served research niche**: most interpretability tooling assumes CUDA clusters. Making rigorous mapping feasible on a Mac is part of the contribution.537538Document (without necessarily implementing) how each pipeline stage would map to x86 + discrete GPU.539540### Stretch target541542```text543A full, causally-verified, versioned atlas of a 7B–14B open-weight model,544produced end-to-end on a single 32–64 GB Mac,545comparable across ≥2 quantization levels and ≥2 model sizes,546published on modelmap.io with complete provenance.547```548549Do not assume this is achievable. The research must establish the actual limits.550551---552553## 16. What counts as a breakthrough554555A result is scientifically interesting if at least one of the following is demonstrated:556557**A.** A mapping methodology whose claims survive causal verification at a measured, reported rate — with the rate itself published.558**B.** Cheap (weight-only or low-capture) maps that reliably predict expensive properties (capability localization, quantization sensitivity, pruning tolerance, working sets).559**C.** A cross-model coordinate system allowing internal maps to be meaningfully compared across sizes or families.560**D.** The first systematic map of how quantization deforms internal structure of local models.561**E.** A demonstrated bridge to systems research: maps that materially improve out-of-core execution decisions in `localvm-research`.562**F.** A rigorous negative result: demonstration that a widely-used mapping technique fails replication or causal verification under controlled conditions.563564## 17. Failure criteria565566Be willing to conclude that an approach does not work. For example:567568```text569probe results are dominated by seed/dataset noise570localized structure evaporates under causal testing571SAE features do not replicate across training runs572maps are quantization-fragile to the point of uselessness573cross-model alignment finds no stable correspondence574capture costs exceed consumer-Mac feasibility at useful model sizes575method agreement is near chance576```577578If these occur, document them in `research/LOG.md` and the relevant `analysis.md`, publish the negative result where warranted, and pivot.579580---581582## 18. Deliverables583584```text585README.md, CITATION.cff586research/  (state_of_the_art, research_gaps, candidate_ranking,587            methodology, novelty_check, LOG, bibliography)588src/       (the modelmap library)589experiments/, benchmarks/, results/590atlas/     (versioned map artifacts with provenance + confidence)591site/      (modelmap.io source)592tools/, docs/593```594595The final README must explain: the problem; what existing interpretability work does; what gap was found; the methodology; why it is sound; experimental evidence (numbers, named Mac hardware); the atlas and site; limitations; exact Mac reproduction instructions; future research.596597---598599## 19. Phase 11 — Novelty verification600601Before claiming novelty of any methodology, map type, or finding, perform a dedicated final literature search using terminology derived from what was actually built — and every plausible synonym and conceptual equivalent (e.g., "model atlas", "neuron catalog", "feature browser", "representation cartography", "interpretability benchmark", "quantized-model interpretability"). Assume an idea is **not** novel until evidence suggests otherwise. Record process and conclusion in `research/novelty_check.md`.602603---604605## 20. Most important instruction606607Do not become attached to any particular technique suggested in this document.608609The purpose of this project is **not to implement a preconceived SAE browser, neuron catalog, or circuit viewer**. Those ideas are merely clues.610611The actual assignment is:612613> **Search deeply enough, reason independently enough, control and replicate rigorously enough, and intervene causally enough to discover what can truly be known about the inside of a local LLM on consumer Mac hardware — and publish only that, at its honest confidence level, on modelmap.io.**614615Start with literature. Then generate hypotheses. Then falsify them experimentally. Let evidence determine the methodology.616617The ideal outcome is not another gallery of suggestive visualizations. The ideal outcome is a **reproducible cartographic standard for open-weight models** — where every map is versioned, provenanced, confidence-labeled, and regenerable by anyone with a Mac — turning:618619```text620"we think the model does X"621```622623into:624625```text626"here is the map, its evidence level, and the script that rebuilds it"627```628629---630631*Author: Simon-Pierre Boucher — contact@spboucher.ai — https://modelmap.io — All research artifacts in this repository carry this attribution.*632