# project: modelmap document: Phase 1 notes — §4.7 Cross-model and cross-scale comparison author: Simon-Pierre Boucher contact: contact@spboucher.ai website: https://modelmap.io created: 2026-08-12 status: draft

# §4.7 — Cross-model and cross-scale comparison

# A1. CKA / SVCCA / RSA as similarity metrics for LLMs

• Key papers: SVCCA: https://arxiv.org/abs/1706.05806 ; CKA: https://arxiv.org/abs/1905.00414 ; survey: Klabunde et al. 2023 (rev. 2025), https://arxiv.org/abs/2305.06329 ; ReSi benchmark (24 measures, ICLR 2025): https://arxiv.org/abs/2408.00531 ; LLM-specific: https://arxiv.org/abs/2312.02730
• Pathologies: outlier-direction sensitivity (Davari et al., https://arxiv.org/abs/2210.16156); score saturation for LLM pairs (most non-Llama pairs >0.8); naive estimator drifts to 1 as feature/sample ratio grows → false alignment discoveries. ReSi: different measures rank model pairs inconsistently — metric choice is itself a confounder, must be part of provenance.
• Cost: cheap (minibatch-CKA on cached activations, NumPy/Accelerate/MLX). Fully feasible 0.5B–14B.
• Epistemic status: Established tool; contested as evidence — every convergence claim needs baseline calibration (A6).

# A2. Universal neurons across seeds (Gurnee et al.)

• Key papers: https://arxiv.org/abs/2401.12181 (2024); follow-up with ablation-based functional impact (2025): https://arxiv.org/abs/2508.00903
• Claims: only 1–5% of MLP neurons are universal across independently seeded GPT-2 runs (activation correlation over 100M tokens); these are disproportionately interpretable (unigram, alphabet, position, entropy-regulation families). Negative half: ~95% of neurons are seed-specific — single-neuron catalogs are mostly noise.
• Methodology template for modelmap: correlation-over-corpus → taxonomize → ablate.
• Cost: paired capture over ~10⁸ tokens — heavy but streamable; feasible ≤1B pairs on 32–64 GB with disk-backed store (expH regime).
• Epistemic status: Established (replicated, causally confirmed).

# A3. Universality of induction heads

• Key papers: Olsson et al. 2022, https://arxiv.org/abs/2209.11895 ; ICL essentiality: https://arxiv.org/abs/2407.07011 ; cross-architecture (Transformer vs Mamba): https://arxiv.org/abs/2410.06672 ; https://arxiv.org/abs/2411.07071
• Substance: strongest known case of circuit universality — appears in essentially every transformer LM, and analogues in Mamba. A natural "known landmark" for calibrating any cross-model coordinate system: if an alignment method can't match induction heads across two models, it fails the sanity check.
• Cost: cheap — one forward pass with attention capture; feasible to 32B quantized via MLX.
• Epistemic status: Established (multi-team, cross-architecture).

# A4. Cross-scale circuit evolution

• Key papers: Lieberum et al. 2023 (Chinchilla 70B), https://arxiv.org/abs/2307.09458 ; Tigges et al. 2024 (Pythia 70M–2.8B across checkpoints): https://arxiv.org/abs/2407.10827
• Substance: circuit-analysis methods scale to 70B but head-level semantics get muddier; task circuits emerge at consistent token budgets and algorithms stay roughly stable while component identities drift — supports longitudinal/cross-scale maps (Experiment G). The Tigges regime (70M–2.8B, full patching) is comfortably Mac-feasible — the right scale band for expG.
• Epistemic status: Established small-scale; partially established at 70B.

# A5. Feature universality via SAEs; universal SAEs; crosscoder diffing

• Key papers: Lan et al. 2024/2025, https://arxiv.org/abs/2410.06981 ; universal SAEs: Thasarathan et al. 2025, https://arxiv.org/abs/2502.03714 ; Anthropic model diffing: https://transformer-circuits.pub/2024/model-diffing/index.html , https://transformer-circuits.pub/2025/crosscoder-diffing-update/index.html ; sparsity artifacts: Minder et al. 2025, https://arxiv.org/abs/2504.02922 ; open replication: https://www.lesswrong.com/posts/srt6JXsRMtmqAJavD/
• Substance: moderate feature-space overlap across families, strongest mid-layers and concrete concepts. Caveat chain: SAE features are seed-unstable (~30% overlap — see §4.8); crosscoder "model-exclusive" latents partly sparsity artifacts. Any cross-model feature atlas must control for both.
• Cost: SAE training is the expensive item (10⁸–10⁹ activations/layer); ≤3B feasible; crosscoders double storage. SAELens MPS throughput unbenchmarked — genuine expH deliverable.
• Epistemic status: Contested/emerging — real signal, magnitude method- and seed-dependent.

# A6. Platonic Representation Hypothesis and its deflation

• Key papers: Huh et al., ICML 2024, https://arxiv.org/abs/2405.07987 ; linear-net proof: https://arxiv.org/abs/2507.01098 ; Aristotelian critique (2026): https://arxiv.org/abs/2602.14486 ; causes/consequences: https://arxiv.org/abs/2505.13899
• The load-bearing negative result: after calibrating for width/depth confounders, global convergence metrics largely evaporate (partly a look-elsewhere effect from reporting max similarity over many layer pairs). What survives is local neighborhood/topological alignment.
• Practical rule for the atlas: cross-model claims use calibrated, local metrics with null baselines from random/width-matched models.
• Epistemic status: Contested; global-metric version debunked-in-part (2026); local version live.

# A7. Model stitching as functional similarity

• Key papers: Bansal et al. 2021, https://arxiv.org/abs/2106.07682 ; https://arxiv.org/abs/2303.11277 ; "How Not to Stitch" (2024): https://arxiv.org/abs/2412.11299 ; relative representations line: https://arxiv.org/abs/2406.11014
• Substance: low CKA can stitch perfectly and vice versa; results depend heavily on connector expressivity/protocol (affine stitch can "succeed" by relearning); LLM pairs collide with tokenizer mismatch (A9).
• Epistemic status: Established as probe; contested as measure.

# A8. Rosetta Neurons (vision precedent)

• Key paper: Dravid et al., ICCV 2023, https://arxiv.org/abs/2306.09346 (code https://github.com/yossigandelsman/rosetta_neurons)
• Substance: activation-correlation mining finds shared-concept units across 8 heterogeneous vision models without supervision. The matching algorithm (normalized cross-correlation over a shared stimulus set) transfers directly to LLMs with a shared prompt corpus — a concrete candidate for modelmap's cross-model coordinate system. Untested at LLM scale (gap).

# A9. Tokenizer effects on comparability

• Key papers: TokSuite (2025, models identical except tokenizer): https://arxiv.org/pdf/2512.20757 ; cross-tokenizer distillation: https://arxiv.org/abs/2503.20083 ; tokenisation NP-complete: https://arxiv.org/abs/2412.15210
• Substance: no standard solution for aligning activations across tokenizers; workarounds (shared-substring boundaries, word/byte pooling, prompt-level mean pooling) each destroy information. Rule: within-family cross-scale comparison (shared tokenizer) is the safe default; cross-family maps must document the alignment convention in the map schema.
• Epistemic status: Established confound; unsolved methodology (gap).

# A10. Quantization effects on internal representations — the thin shelf (project bet confirmed)

As of 2026-08-12 this literature is ~5 papers, recent, shallow — no systematic multi-technique internal atlas across quantization levels:

• Interpreting the Effects of Quantization on LLMs (IJCNLP-AACL 2025): https://arxiv.org/abs/2508.16785 — 4/8-bit vs FP16; breadth-first, small depth.
• Through a Compressed Lens (2025): https://arxiv.org/abs/2505.13963 — GPTQ/AWQ on Qwen2.5 7B–72B; effects inconsistent in direction, dependent on method × technique × protocol.
• How Quantization Changes Interpretable Features: An SAE Analysis (2026): https://arxiv.org/html/2606.03002v1 — first SAE-feature-drift-under-quantization study; closest existing work to expF; single-method, worth replicating and extending.
• The Illusion of Equivalency (2026): https://arxiv.org/pdf/2607.08734 — benchmark-equivalence masks internal distributional divergence.
• Behavioral-only adjacent: long-context degradation https://arxiv.org/pdf/2505.20276

Verified absent: probe-drift-across-Q-levels with controls; circuit survival under quantization; Q2/Q3 extreme-quantization internals; GGUF/llama.cpp-runtime introspection; weight-geometry-predicts-quantization-sensitivity. expF sits in genuinely open territory — and quantized models are precisely what runs on the target hardware. MLX ships 4/8-bit natively; FP16/Q8/Q4 triplets fit sequentially in 32–64 GB up to ~14B.

• Epistemic status: Emerging; the field's own words ("inconsistent", "illusion of equivalency") mark it unsettled.