# CLAUDE.md

Project codename: modelmap — Internal Cartography of Local Large Language Models Public site: https://modelmap.io (the living research atlas & documentation site) Principal investigator / Author: Simon-Pierre Boucher — contact@spboucher.ai Primary platform: Apple Silicon Mac (macOS 14+) Sister project: localvm-research (out-of-core execution) — findings must cross-pollinate Document status: Living research charter. Claude must treat this as the authoritative project specification.

# 0. Administrative conventions (MANDATORY — read before anything else)

# 0.1 Author header requirement

Every single source file created in this project — without exception — must begin with a standardized author header.

This applies to: Python, C++, Swift, Objective-C, Metal shaders, JavaScript/TypeScript (for the modelmap.io site), shell scripts, Makefiles, CMake files, and any configuration file that supports comments.

# Python / shell / YAML / TOML header

# =============================================================================
#  Project   : modelmap
#  File      : <relative/path/to/file.py>
#  Purpose   : <one-line description of what this file does>
#  Author    : Simon-Pierre Boucher
#  Contact   : contact@spboucher.ai
#  Website   : https://modelmap.io
#  Created   : <YYYY-MM-DD>
#  Modified  : <YYYY-MM-DD>
#  Platform  : macOS / Apple Silicon (arm64)
#  License   : All rights reserved (research code)
# =============================================================================

# C++ / Metal / Swift / JS / TS header

// ============================================================================
//  Project   : modelmap
//  File      : <relative/path/to/file>
//  Purpose   : <one-line description>
//  Author    : Simon-Pierre Boucher
//  Contact   : contact@spboucher.ai
//  Website   : https://modelmap.io
//  Created   : <YYYY-MM-DD>
//  Modified  : <YYYY-MM-DD>
//  Platform  : macOS / Apple Silicon (arm64) — MLX / Metal / MPS
//  License   : All rights reserved (research code)
// ============================================================================

# Markdown research documents (front matter)

---
project: modelmap
document: <name>
author: Simon-Pierre Boucher
contact: contact@spboucher.ai
website: https://modelmap.io
created: <YYYY-MM-DD>
status: draft | reviewed | final
---

Rules:

1. The header must be the first content of the file (after a shebang if present).
2. Modified must be updated on every substantial change.
3. Write tools/check_headers.py early; it must fail if any tracked source file lacks a conforming header, and it runs before every commit.
4. Generated artifacts (maps, JSON atlases, plots) must embed "author": "Simon-Pierre Boucher", "contact": "contact@spboucher.ai", "website": "https://modelmap.io" in their metadata wherever the format permits.
5. Every published map on modelmap.io must carry a visible attribution footer.

# 0.2 macOS-first design constraint

Everything must be designed to run on a Mac. Specifically:

• Target machine: Apple Silicon (M1–M4 family), 16–64 GB unified memory, internal NVMe SSD.
• Default compute paths: MLX, PyTorch with MPS backend, NumPy/Accelerate, custom Metal kernels where hooks/probes need speed. CUDA is allowed only as an isolated optional validation path (src/backends/cuda_optional/), never a core dependency.
• Activation capture at scale generates large tensors: design the capture pipeline for unified memory limits (streaming to disk, memory-mapped activation stores, chunked capture) — never assume activations fit in RAM.
• Storage of maps/activations must respect APFS behavior and Apple NVMe characteristics; use efficient columnar/array formats (safetensors, zarr, parquet, HDF5) benchmarked on macOS, not assumed from Linux numbers.
• Instrumentation must use macOS-native sources: mach task_info, vm_stat, fs_usage, powermetrics (graceful degradation without sudo), sysctl hw.*.
• Build tooling: uv/pip + venv for Python; cmake + AppleClang for C++; xcrun -sdk macosx metal for shaders; standard Node toolchain for the modelmap.io site (static-site friendly — the site must build and preview locally on the Mac).
• A hardware manifest (benchmarks/hardware_manifest.py) records chip, P/E core counts, GPU cores, RAM, SSD model, macOS version, and all software versions into every result and every published map.

# 0.3 Repository discipline

• Git from day one; meaningful commits; no result reported from an uncommitted tree.
• Every map must be reproducible from: commit hash + config file + model hash + seed + hardware manifest.
• Python: type hints, ruff, pytest (numerical probes get correctness tests against tiny reference models).
• Datasets of prompts used for mapping are versioned artifacts with checksums.
• Large binary artifacts (activation dumps) live outside git (local artifact store with an index file committed to git).

# 1. Mission

Your objective is to investigate, design, implement, and experimentally validate a systematic methodology — and the tooling behind it — to discover, measure, and map the internal structure of pretrained open-weight LLMs running locally on consumer Apple Silicon hardware, and to publish the resulting maps as a rigorous, reproducible public atlas at modelmap.io.

The target problem is not:

• training new models;
• fine-tuning models to change their behavior;
• reproducing an existing interpretability library (TransformerLens, NNsight, SAELens, Captum) without adding anything;
• producing pretty but unfalsifiable visualizations;
• benchmarking model outputs (that is leaderboard work, not cartography);
• speculative "the model thinks X" storytelling without measurement.

The target is broader and more rigorous:

Given an already-trained open-weight model, determine what can actually be known, measured, localized, and mapped about its internal organization — where knowledge lives, how computation is distributed, which structures are stable across inputs, layers, scales, and model families — using only local, consumer-grade Mac hardware, and turn that knowledge into reproducible, comparable, publishable maps.

Do not assume in advance which mapping techniques are informative. Do not force a particular interpretability framework. Do not begin implementation before understanding the state of the art. Your job is to discover what can genuinely be mapped — and what cannot.

# 2. Core research questions

Investigate:

Q1 — Localization. Where inside a pretrained LLM do specific capabilities, knowledge domains, languages, and behaviors reside? Are they localized (layers, heads, neurons, weight blocks, feature directions) or diffuse?

Q2 — Structure. What stable internal structures exist across inputs — circuits, feature directions, attention patterns, activation manifolds, weight-space geometry — and at what granularity are they real rather than artifacts of the probing method?

Q3 — Comparability. Can internal maps be compared across model sizes, checkpoints, quantization levels, and model families? Is there a common "coordinate system" for model internals?

Q4 — Cost. Which mapping techniques are feasible on a 16–64 GB Mac, at which model sizes, and what is the accuracy/cost frontier of local interpretability?

Q5 — Utility. Do the maps predict anything useful — quantization sensitivity, pruning tolerance, working-set behavior (feeding localvm-research), failure modes, editing targets?

The most important distinction to keep sharp throughout:

what a probing method reports
≠
what the model actually computes
≠
what is stable across methods, seeds, and datasets
≠
what is causally verified by intervention

A claim only enters the atlas at the confidence level its evidence supports. Correlational maps and causally-verified maps must never be visually or textually conflated.

# Guiding principle

Do not optimize first for spectacular visuals. Initially optimize for discovering which measurements are real — reproducible, method-robust, causally grounded.

Negative results ("technique X produces maps that do not replicate across seeds") are first-class publishable findings on modelmap.io.

# 3. Project structure (real research-project layout)

Create this skeleton (with header-compliant placeholder files) before Phase 1 concludes:

modelmap/
├── CLAUDE.md                      # this charter
├── README.md                      # public summary (written last, updated continuously)
├── CITATION.cff                   # author: Simon-Pierre Boucher
├── LICENSE
├── pyproject.toml
├── Makefile                       # setup, lint, test, capture, map, site, headers
│
├── research/                      # the scientific paper trail
│   ├── LOG.md                     # dated, append-only research log
│   ├── state_of_the_art.md       # Phase 2 deliverable
│   ├── research_gaps.md          # Phase 3 deliverable
│   ├── candidate_ranking.md      # Phase 4 deliverable
│   ├── methodology.md            # the formal mapping methodology (Phase 9)
│   ├── novelty_check.md          # Phase 11 deliverable
│   ├── bibliography.md           # every source, links + access dates
│   └── notes/                     # per-topic reading notes
│
├── src/
│   └── modelmap/
│       ├── __init__.py
│       ├── models/                # loading (safetensors/GGUF), architecture introspection
│       ├── capture/               # activation hooks, streaming capture, mmap stores
│       ├── probes/                # linear probes, logit lens, tuned lens, causal probes
│       ├── interventions/         # ablation, patching, steering, weight zeroing
│       ├── features/              # SAE training/inference, dictionary methods
│       ├── geometry/              # weight-space analysis: SVD, spectra, similarity (CKA…)
│       ├── graphs/                # circuit extraction, attribution graphs
│       ├── atlas/                 # map schema, serialization, versioning, provenance
│       ├── viz/                   # figure + interactive-map generation
│       ├── backends/              # mlx_backend / mps_backend / cpu_backend / cuda_optional
│       ├── instrumentation/       # macOS-native cost measurement
│       └── stats/                 # replication tests, bootstrap CIs, multiple-comparison control
│
├── experiments/
│   ├── micro/
│   │   ├── expA_probe_reliability/
│   │   ├── expB_localization_vs_diffusion/
│   │   ├── expC_causal_verification/
│   │   ├── expD_cross_input_stability/
│   │   ├── expE_weight_geometry/
│   │   ├── expF_quantization_map_drift/
│   │   ├── expG_cross_model_alignment/
│   │   └── expH_capture_cost_frontier/
│   ├── candidate_01/
│   ├── candidate_02/
│   └── candidate_03/
│       └── (each: README.md, hypothesis.md, implementation/,
│                benchmark.py, results/, analysis.md)
│
├── atlas/                         # the maps themselves (versioned data products)
│   └── <model_id>/<map_type>/<version>/
│       ├── map.json / map.zarr
│       ├── provenance.json        # commit, config, model hash, hardware, dates
│       └── confidence.md          # evidence level: correlational | replicated | causal
│
├── site/                          # modelmap.io source (static-site, builds locally on Mac)
│   ├── content/                   # one page per model, per map type, per finding
│   ├── components/                # interactive map viewers
│   └── data/                      # published atlas exports
│
├── benchmarks/
│   ├── harness.py
│   ├── hardware_manifest.py
│   └── promptsets/                # versioned mapping corpora: code, math, FR, EN, facts…
│
├── results/                       # raw + aggregated results
│   └── <experiment_id>/<timestamp>/
│
├── tools/
│   ├── check_headers.py
│   ├── new_experiment.py          # scaffolds compliant experiment dirs
│   ├── new_map.py                 # scaffolds a compliant atlas entry with provenance
│   └── publish.py                 # atlas → site/data export with attribution
│
└── docs/

Every experiment directory is scaffolded by tools/new_experiment.py with the seven-field scientific block (§10). Every atlas entry is scaffolded by tools/new_map.py and cannot be published without a completed provenance.json and confidence.md.

# 4. Phase 1 — Ultra-deep literature research

Before proposing a methodology, perform an extremely deep search of current literature, repositories, technical reports, papers, preprints, blog posts, implementations, and discussions.

Search arXiv, OpenReview, conference proceedings (NeurIPS, ICML, ICLR, ACL, EMNLP), the mechanistic-interpretability community corpus (Anthropic interpretability publications, Transformer Circuits thread, Neel Nanda's work, EleutherAI, LessWrong/Alignment Forum technical posts), GitHub (TransformerLens, NNsight, SAELens, Penzai, Captum, baukit), Hugging Face, and Apple/MLX engineering material.

Prefer primary sources. For every important technique find the paper and the implementation, and note whether it runs on Apple Silicon. Log everything in research/bibliography.md with URLs and access dates. Follow citations backward and forward.

# Areas that must be investigated

# 4.1 Observational probing

Linear probes; logit lens; tuned lens; early decoding; representation reading; concept erasure (LEACE-style); probing classifier pitfalls and controls; selectivity/controls literature (what probes overfit to).

# 4.2 Causal / interventional methods

Activation patching; path patching; attribution patching; causal tracing (ROME-style locating); ablations (zero, mean, resample); causal scrubbing; interchange interventions; steering vectors; representation engineering; distributed alignment search.

# 4.3 Feature decomposition

Sparse autoencoders (SAEs) and their variants (gated, top-k, jumpReLU, crosscoders, transcoders); dictionary learning; superposition theory; polysemanticity; feature splitting; SAE evaluation problems (reconstruction vs interpretability vs faithfulness); automated feature labeling and its failure modes.

# 4.4 Circuits and computational graphs

Induction heads; IOI-style circuit analyses; attention head taxonomies; MLP key–value memory view; attribution graphs; automated circuit discovery (ACDC and successors); edge/node pruning of computational graphs; faithfulness metrics for circuits.

# 4.5 Weight-space cartography (no forward pass required)

SVD spectra of weight matrices; effective rank per layer; weight statistics across depth; outlier channels/dimensions; cross-layer similarity (CKA, CCA, Procrustes); model stitching; permutation alignment / git re-basin; mode connectivity; delta analysis between checkpoints and between base/instruct variants; MoE router and expert structure.

# 4.6 Knowledge localization and editing

ROME, MEMIT and successors; where facts are stored vs where they are retrieved; the localization-vs-editing critique literature (does editing success actually imply localization?); knowledge neurons; multilingual knowledge sharing.

# 4.7 Cross-model and cross-scale comparison

Representation similarity across scales and families; universality of features/circuits; platonic representation hypothesis debates; tokenizer effects on comparability; quantization effects on internal representations (largely unmapped — note carefully).

# 4.8 Statistics and epistemology of interpretability

Illusions in interpretability results; seed sensitivity; dataset sensitivity of probes; multiple-comparison problems when scanning thousands of neurons/heads; pre-registration analogues; faithfulness vs plausibility; benchmarking interpretability methods (e.g., finding planted circuits in known models).

# 4.9 Systems side (local mapping at scale)

Activation caching formats and costs; hook overhead in MLX vs PyTorch-MPS; streaming SAE training on limited memory; disk layouts for random access into activation stores; what interpretability workloads exist for GGUF/quantized runtimes (llama.cpp introspection); GPU capture on Metal.

# 4.10 Visualization and atlas design

Prior art on model atlases (Neuroscope, Neuronpedia, OpenAI microscope-style efforts, feature browsers); what made them useful or misleading; provenance and versioning practices for data products; how to encode uncertainty visually.

# 5. Phase 2 — State-of-the-art map

Produce research/state_of_the_art.md. Organize every relevant technique by:

Technique
What it claims to reveal
Observational or causal?
Model access required (weights / activations / gradients)
Compute + memory cost class
Feasible on 16–64 GB Apple Silicon? (measured or estimated, at which model sizes)
Known failure modes / illusions
Reproducibility record
Open-source implementation (and does it build on macOS arm64 / MPS / MLX?)
Main limitation
Opportunity for extension

Do not simply summarize papers. Identify overlaps, already-tried combinations, and ideas that seem novel but are known. Explicitly flag the epistemic status of each technique (established / contested / debunked-in-part).

# 6. Phase 3 — Identify genuine gaps

Produce research/research_gaps.md. For every promising gap explain:

1. what existing work does;
2. what it does not do;
3. why the missing capability might matter;
4. whether there is a plausible mathematical or empirical reason it could work;
5. the biggest reason it might fail;
6. the smallest experiment capable of falsifying it — runnable on a Mac.

Generate at least 20 substantially different candidate directions. Do not make them superficial variants.

Example categories (non-binding): quantization-aware internal maps; weight-only cheap cartography as a pre-screen for expensive activation methods; cross-model coordinate systems; maps that predict systems properties (working sets, pruning tolerance — bridge to localvm); method-agreement scoring (map = intersection of techniques); longitudinal maps across a model family; multilingual localization atlases; local-hardware-feasible SAE recipes; causal-verification pipelines that scale down; uncertainty-first atlas schemas.

These are examples only. Do not constrain the search to them.

# 7. Phase 4 — Rank candidate approaches

Create research/candidate_ranking.md. Score every idea 1–10 with written reasoning on:

Scientific value (what new knowledge does the map contain?)
Epistemic soundness (can claims be causally verified?)
Novelty
Feasibility on Apple Silicon (memory, time, kernel availability)
Reproducibility potential
Cross-model generality
Public atlas value (is it worth publishing on modelmap.io?)
Implementation complexity
Synergy with localvm-research
Risk

Select roughly 3–5 strongest candidates for prototyping.

Prefer directions that produce comparable, versionable, falsifiable map artifacts rather than one-off analyses.

# 8. Phase 5 — Experimental framework

Build the framework before mapping large models. Correctness and measurement first; optimization later.

Preferred stack (Mac-first):

Python (MLX, PyTorch-MPS, NumPy, safetensors, zarr) for research tooling
C++ / Metal where hook or SAE throughput demands it
TypeScript + static-site generator for modelmap.io (local build/preview on the Mac)
CUDA only as isolated optional validation

# 8.1 Model progression

0.5B–1B  →  3B  →  7B–8B  →  14B  →  32B

Move larger only when evidence and the cost frontier (Experiment H) support it. Prefer modern open-weight families with multiple sizes (to enable cross-scale maps) and both base + instruct checkpoints (to enable delta maps).

# 8.2 Instrumentation

Measure, for every mapping run:

resident/peak RAM              activation bytes captured
disk bytes written/read        capture overhead vs plain inference
wall-clock per mapped unit     GPU/CPU utilization (P vs E cores)
hook dispatch overhead         SAE training throughput
thermal state / throttling     energy via powermetrics when available

And for every scientific result:

replication across ≥3 seeds        replication across ≥2 prompt sets
bootstrap confidence intervals      effect sizes, not just p-values
method agreement (≥2 techniques)    causal verification status
multiple-comparison correction when scanning many units

# 8.3 Most important scientific metric

Track, for every map:

REPLICATION RATE (does the map reproduce under resampling of seeds/data?)
CAUSAL CONFIRMATION RATE (what fraction of localized claims survive intervention?)

A beautiful map that does not replicate is worth less than a boring one that does.

# 8.4 Baselines and controls

Every mapping technique must be run with controls: shuffled labels for probes; random-direction baselines for steering; randomly-initialized-model baselines where meaningful; resample ablations vs zero ablations. Never claim structure without showing the null.

# 9. Phase 6 — Micro-experiments

Each lives in experiments/micro/ with the standard scaffold.

# Experiment A — Probe reliability

For a small model, train linear probes for a set of properties across all layers, with shuffled-label controls, ≥5 seeds, ≥2 datasets. Quantify how much probe accuracy varies with seed/data. Establish the noise floor every later map must exceed.

# Experiment B — Localization vs diffusion

For selected capabilities (a language, a knowledge domain, arithmetic, code syntax), measure how concentrated the supporting signal is across layers/heads/neurons/weight blocks. Ask: does removing the top-k localized units remove the capability, or does it survive (diffusion)? Test across code, math, chat, French, English, factual recall, reasoning.

# Experiment C — Causal verification pipeline

Take the top findings from A/B and verify them with activation patching and ablation. Measure the correlational→causal survival rate. This number calibrates all future confidence labels in the atlas.

# Experiment D — Cross-input stability

For fixed internal structures (heads, features, directions), measure stability across prompts, domains, context lengths, and sampling temperature. Compute overlap/Jaccard of "important units" across inputs. Determine which map types are input-conditional vs input-invariant.

# Experiment E — Weight-space geometry (zero-forward-pass maps)

Compute per-layer SVD spectra, effective ranks, outlier dimensions, and cross-layer CKA purely from weights. Test whether these cheap maps predict anything measured in A–D. If yes, weight-only cartography becomes a fast pre-screen for any model.

# Experiment F — Quantization map drift

Map the same model at FP16, Q8, Q4, Q2 (where practical). Measure how much probes, feature directions, and localized circuits drift under quantization. This is under-explored territory and directly relevant to local models, which are almost always quantized.

# Experiment G — Cross-model alignment

For two sizes of the same family, attempt to align internal maps (representation similarity, feature matching, permutation alignment). Determine whether a shared coordinate system is achievable and at what granularity.

# Experiment H — Capture cost frontier (macOS-specific)

Measure the real cost of activation capture on the Mac: hook overhead in MLX vs PyTorch-MPS, streaming write throughput to the internal NVMe, storage formats (safetensors vs zarr vs raw mmap), maximum feasible capture per model size within 16/32/64 GB. Produce a published "what can you map on which Mac" table — itself a modelmap.io artifact.

# 10. Scientific discipline

For every experiment write, in hypothesis.md / analysis.md:

Hypothesis
Falsification criterion
Method (including controls)
Baseline / null
Result (with effect sizes and CIs)
Interpretation (with explicit confidence level)
Next experiment

Avoid confirmation bias. If a technique produces illusory structure, record it — that is a finding. Do not silently discard failures.

# Evidence standard

Never write "this neuron does X", "this layer stores Y", "feature Z represents W" without stating: the method, the controls, the replication status, and whether the claim is correlational or causally verified. Report: number of seeds, prompt sets, effect sizes, CIs, hardware, model+quantization, software versions.

# Confidence taxonomy (used everywhere, including on the site)

Level 0 — anecdotal        (single run, no controls; never published alone)
Level 1 — correlational    (controlled, replicated ≥3 seeds, ≥2 datasets)
Level 2 — method-robust    (Level 1 + agreement across ≥2 independent techniques)
Level 3 — causal           (Level 2 + intervention confirms the claim)

Every atlas entry carries its level in confidence.md and visibly on modelmap.io.

# 11. Phase 7 — Prototype candidate methodologies

For each of the strongest candidates create experiments/candidate_0N/ with the standard contents. All implementations must run on the primary Mac target without CUDA. Each candidate's analysis.md must end with an explicit recommendation: promote to atlas pipeline, iterate, or reject (with reasons).

# 12. Phase 8 — Automatic research loop

Operate as a research agent. After every experiment: analyze; pick the most informative next experiment; re-search the literature when results surprise; update hypotheses; modify the methodology; rerun; compare; continue until evidence strongly favors or rejects the approach. Do not follow a rigid roadmap if experiments contradict it.

Maintain research/LOG.md (append-only): date/time with timezone, question, experiment, result, interpretation, decision. The entire reasoning process must be auditable.

# 13. Phase 9 — Formalize the mapping methodology

Once micro-experiments and candidates converge, write research/methodology.md: the formal, reusable pipeline that turns (model checkpoint + prompt corpora + budget) into (versioned atlas entries with confidence levels).

It must specify: capture protocol; probe/feature/circuit procedures actually retained; mandatory controls; replication requirements; causal-verification requirements per confidence level; cost model per Mac tier; map schema; provenance schema; publication checklist.

The methodology is itself a primary deliverable — it should let any researcher with a Mac reproduce a map.

# 14. Phase 10 — Build the atlas pipeline and modelmap.io

Build the production pipeline supported by evidence. A possible CLI shape (finalize only after research):

modelmap capture MODEL_PATH --promptset benchmarks/promptsets/core --out atlas/
modelmap analyze atlas/<model_id> --maps probes,features,geometry,circuits
modelmap verify  atlas/<model_id> --level causal
modelmap publish atlas/<model_id> --site site/

# The site (modelmap.io)

• Static-site architecture, built and previewed locally on the Mac; every page generated from atlas data, never hand-edited numbers.
• One page per model; per map type; interactive viewers (layer/head/feature browsers) built from atlas/ exports.
• Every visualization displays: confidence level, provenance (commit, model hash, date, hardware), and how to reproduce.
• A methodology section publishing research/methodology.md.
• A negative-results section. Publishing what does not replicate is part of the mission.
• Attribution footer on every page: Simon-Pierre Boucher — contact@spboucher.ai.

# Hard constraints

• Only open-weight models are mapped; the original checkpoint is always the object of study.
• No claim appears on modelmap.io above its evidence level.
• Nothing is published that cannot be regenerated from committed code + versioned data.

# 15. Target hardware

Primary research target:

Apple Silicon Mac (laptop or desktop)
16–64 GB unified memory
internal Apple NVMe SSD
Metal-capable GPU sharing memory with CPU

Apple Silicon matters here because: unified memory lets capture pipelines avoid device copies; MLX enables custom, low-overhead hooks; fast internal SSDs make large activation stores workable; and — crucially — local interpretability on consumer hardware is itself an under-served research niche: most interpretability tooling assumes CUDA clusters. Making rigorous mapping feasible on a Mac is part of the contribution.

Document (without necessarily implementing) how each pipeline stage would map to x86 + discrete GPU.

# Stretch target

A full, causally-verified, versioned atlas of a 7B–14B open-weight model,
produced end-to-end on a single 32–64 GB Mac,
comparable across ≥2 quantization levels and ≥2 model sizes,
published on modelmap.io with complete provenance.

Do not assume this is achievable. The research must establish the actual limits.

# 16. What counts as a breakthrough

A result is scientifically interesting if at least one of the following is demonstrated:

A. A mapping methodology whose claims survive causal verification at a measured, reported rate — with the rate itself published. B. Cheap (weight-only or low-capture) maps that reliably predict expensive properties (capability localization, quantization sensitivity, pruning tolerance, working sets). C. A cross-model coordinate system allowing internal maps to be meaningfully compared across sizes or families. D. The first systematic map of how quantization deforms internal structure of local models. E. A demonstrated bridge to systems research: maps that materially improve out-of-core execution decisions in localvm-research. F. A rigorous negative result: demonstration that a widely-used mapping technique fails replication or causal verification under controlled conditions.

# 17. Failure criteria

Be willing to conclude that an approach does not work. For example:

probe results are dominated by seed/dataset noise
localized structure evaporates under causal testing
SAE features do not replicate across training runs
maps are quantization-fragile to the point of uselessness
cross-model alignment finds no stable correspondence
capture costs exceed consumer-Mac feasibility at useful model sizes
method agreement is near chance

If these occur, document them in research/LOG.md and the relevant analysis.md, publish the negative result where warranted, and pivot.

# 18. Deliverables

README.md, CITATION.cff
research/  (state_of_the_art, research_gaps, candidate_ranking,
            methodology, novelty_check, LOG, bibliography)
src/       (the modelmap library)
experiments/, benchmarks/, results/
atlas/     (versioned map artifacts with provenance + confidence)
site/      (modelmap.io source)
tools/, docs/

The final README must explain: the problem; what existing interpretability work does; what gap was found; the methodology; why it is sound; experimental evidence (numbers, named Mac hardware); the atlas and site; limitations; exact Mac reproduction instructions; future research.

# 19. Phase 11 — Novelty verification

Before claiming novelty of any methodology, map type, or finding, perform a dedicated final literature search using terminology derived from what was actually built — and every plausible synonym and conceptual equivalent (e.g., "model atlas", "neuron catalog", "feature browser", "representation cartography", "interpretability benchmark", "quantized-model interpretability"). Assume an idea is not novel until evidence suggests otherwise. Record process and conclusion in research/novelty_check.md.

# 20. Most important instruction

Do not become attached to any particular technique suggested in this document.

The purpose of this project is not to implement a preconceived SAE browser, neuron catalog, or circuit viewer. Those ideas are merely clues.

The actual assignment is:

Search deeply enough, reason independently enough, control and replicate rigorously enough, and intervene causally enough to discover what can truly be known about the inside of a local LLM on consumer Mac hardware — and publish only that, at its honest confidence level, on modelmap.io.

Start with literature. Then generate hypotheses. Then falsify them experimentally. Let evidence determine the methodology.

The ideal outcome is not another gallery of suggestive visualizations. The ideal outcome is a reproducible cartographic standard for open-weight models — where every map is versioned, provenanced, confidence-labeled, and regenerable by anyone with a Mac — turning:

"we think the model does X"

into:

"here is the map, its evidence level, and the script that rebuilds it"

Author: Simon-Pierre Boucher — contact@spboucher.ai — https://modelmap.io — All research artifacts in this repository carry this attribution.