spb/modelmap Public License

1---2project: modelmap3document: Phase 1 notes — §4.1 Observational probing4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6website: https://modelmap.io7created: 2026-08-128status: draft9---1011# §4.1 — Observational probing1213## A1. Linear probes (probing classifiers)14- **Key papers:** Alain & Bengio 2016, https://arxiv.org/abs/1610.01644 ; Belinkov, CL 2022, https://arxiv.org/abs/2102.1245215- **Reveals:** whether a property is *linearly decodable* from hidden states per layer — an "information availability" profile.16- **Type:** Observational (decodability ≠ use by the model).17- **Cost:** Cheap — one forward pass per prompt + logistic/ridge on cached activations. Feasible 0.5B–14B on 16–64 GB (14B fp16 ≈ 28 GB weights → 64 GB tier or 4-bit). Main cost = activation storage; stream to disk.18- **Implementations:** TransformerLens `run_with_cache` (MPS opt-in — some ops lack stable MPS kernels: https://transformerlensorg.github.io/TransformerLens/content/getting_started.html); baukit https://github.com/davidbau/baukit (device-agnostic). Trivial in MLX.19- **Failure modes:** probe learns the task (Hewitt & Liang); decodable-but-unused information; dataset/seed sensitivity; high-capacity probes find structure in *random* models.20- **Epistemic status:** Established as decodability measurement; contested as evidence of use.2122## A2. Logit lens23- **Key source:** nostalgebraist 2020, https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens24- **Reveals:** layer-by-layer refinement of the next-token prediction (final LN + unembedding applied to intermediate residual).25- **Cost:** Near-free; ideal Mac technique; trivial in MLX.26- **Failure modes:** works on GPT-2 family, poorly/misleadingly on BLOOM/OPT/some Llamas (intermediate residual bases differ from final — documented in Belrose et al. 2023); early layers unreadable.27- **Epistemic status:** Established quick diagnostic; **debunked-in-part as a quantitative claim** about intermediate beliefs.2829## A3. Tuned lens30- **Key paper:** Belrose et al. 2023, https://arxiv.org/abs/2303.08112 ; code https://github.com/AlignmentResearch/tuned-lens31- **Reveals:** same as logit lens with trained per-layer affine translators — less biased prediction trajectories.32- **Cost:** Moderate (hours on consumer hardware ≤7B; pretrained lenses exist). Model frozen — no backward through model.33- **Failure modes:** translators can "do the work" (mitigated by affine-only capacity); **lens must be retrained per checkpoint AND per quantization level** — directly relevant to Experiment F.34- **Epistemic status:** Established refinement; validated to 20B (Pythia/NeoX/OPT).3536## A4. Early decoding / vocabulary-space successors (DoLa, Future Lens, Patchscopes)37- **Key papers:** DoLa: Chuang et al. 2023, https://arxiv.org/abs/2309.03883 ; Future Lens: Pal et al., CoNLL 2023, https://future.baulab.info/ ; Patchscopes: Ghandeharioun et al., ICML 2024, https://arxiv.org/abs/2401.0610238- **Reveals:** DoLa: which layers hold "factual" signal. Future Lens: hidden states encode ≥2 tokens ahead. Patchscopes: unifying framework — patch a hidden state into another prompt/model and let the LLM verbalize it; subsumes logit/tuned lens and fixes early-layer unreadability.39- **Cost:** Cheap–moderate; all ≤14B on Mac. Patchscopes ≈ 2× inference.40- **Failure modes:** verbalized readouts inherit the model's confabulation; DoLa's premise unreliable across families ("Steering off Course", ACL 2025).41- **Epistemic status:** Established tools; DoLa mechanism contested. Also: SimLens (2025) https://arxiv.org/pdf/2507.176184243## A5. Representation reading / RepE44- **Key paper:** Zou et al. 2023, https://arxiv.org/abs/2310.01405 ; code https://github.com/andyzoujm/representation-engineering ; survey of challenges (2025): https://arxiv.org/pdf/2502.1760145- **Reveals:** population-level "reading vectors" for high-level concepts via PCA on activation differences over stimulus pairs.46- **Cost:** Cheap (dozens–hundreds of forwards + PCA). Very Mac-friendly.47- **Implementations:** repeng https://github.com/vgel/repeng — control vector "in under sixty seconds", works on CPU/MPS, **exports GGUF control vectors for llama.cpp** (PR https://github.com/ggml-org/llama.cpp/pull/5970) — directly relevant to mapping quantized local models.48- **Failure modes:** concept vectors conflate correlated stimulus features; single-direction assumption; LLM-judge evaluation (AxBench: prompting baselines beat most representation methods).49- **Epistemic status:** Linear concept directions established; methodology reliability contested.5051## A6. Concept erasure — INLP, amnesic probing, LEACE52- **Key papers:** INLP: Ravfogel et al., ACL 2020, https://aclanthology.org/2020.acl-main.647.pdf ; amnesic probing: Elazar et al., TACL 2021, https://aclanthology.org/2021.tacl-1.10/ ; LEACE: Belrose et al., NeurIPS 2023, https://arxiv.org/abs/2306.03819 ; code https://github.com/EleutherAI/concept-erasure53- **Reveals:** whether the model *uses* a property: erase it (LEACE: closed-form, provably-minimal-damage linear guarding) and observe behavioral change. Interventional on representations — bridges to §4.2.54- **Cost:** Cheap–moderate; LEACE needs only class-conditional means/covariances. Mac-feasible to 14B.55- **Failure modes:** INLP damages representations broadly (rank loss); linear erasure can leave or *increase* nonlinearly-decodable information; erasure-operator choice changes amnesic conclusions (2025: https://arxiv.org/html/2506.11673). Post-LEACE: Oracle-LEACE https://blog.eleuther.ai/oracle-leace/ ; nonlinear erasure via density matching https://arxiv.org/abs/2507.1234156- **Epistemic status:** LEACE established for the linear guarantee; amnesic-probing inferences contested.5758## A7. Probing pitfalls: control tasks, selectivity, MDL59- **Key papers:** Hewitt & Liang, EMNLP 2019, https://arxiv.org/abs/1909.03368 ; Voita & Titov 2020 (MDL), https://arxiv.org/pdf/2003.1229860- **Substance:** **selectivity** = task accuracy − control-task accuracy; MDL replaces accuracy with codelength. Probes overfit to: word identity/frequency memorization, spurious dataset correlates, annotation artifacts, probe capacity, shared vocabulary between splits.61- **Implication for modelmap (Experiment A is exactly this):** every probe map needs shuffled-label controls, selectivity or MDL reporting, ≥2 datasets, ≥3–5 seeds, randomly-initialized-model baselines.62- **Epistemic status:** Established best practice; ignoring it is the field's canonical failure.63