---
project: modelmap
document: expA_probe_reliability — hypothesis (registered before run)
author: Simon-Pierre Boucher
contact: contact@spboucher.ai
website: https://modelmap.io
created: 2026-08-12
modified: 2026-08-12
status: reviewed
---

# Hypothesis — expA (probe reliability / the noise floor)

> Probe reliability — noise floor of linear probes across seeds and datasets.
> Registered 2026-08-12, before any run. The harness
> (`src/modelmap/probes/linear.py` + `src/modelmap/stats/replication.py`) is
> implemented and unit-tested against planted structure and pure noise;
> no real-model run has happened yet.

```text
Hypothesis              : On a 0.5B-class model, per-layer linear-probe maps
                          for simple properties (language ID, code-vs-prose,
                          arithmetic-context) are seed-stable (probe-accuracy
                          SD across ≥5 training seeds < 2 points) but
                          dataset-SENSITIVE: switching prompt corpus moves
                          per-layer accuracy by more than the seed SD on at
                          least a third of layers — i.e. the dataset, not the
                          probe seed, is the dominant noise source
                          (Bolukbasi-consistent).
Falsification criterion : Dies if seed variance ≥ dataset variance across the
                          board (then seeds dominate and every later map needs
                          more seeds, not more corpora); the harness itself is
                          invalid if selectivity on a random-init model
                          exceeds 0.05 (fake structure).
Method                  : Qwen3-0.6B (or Pythia-410M fallback) via mlx-lm;
                          capture residual stream at every layer over ≥2
                          disjoint prompt sets per property (versioned,
                          checksummed promptsets); probe_with_control per
                          (layer, property, seed) for 5 seeds; report
                          selectivity, bootstrap CIs, BH-FDR across the layer
                          scan; replication_rate on top-k layer sets.
Baseline / null         : shuffled-label control (built into every probe);
                          random-init model of identical architecture;
                          random-direction projections.
Result                  : (pending)
Interpretation          : (pending — with explicit confidence level)
Next experiment         : (pending — expA output becomes the first map card)
```

---

# Hypothesis — expA run #2 (structure-borne properties, token-balanced)

Registered 2026-08-12 **before** the run, after run #1 failed its validity
gate (twin at ceiling on lexically separable classes). Promptsets v2 are
designed so no token distribution separates the classes:

- **word_order** — grammatical sentence vs seeded scramble of the SAME words
  (perfectly token-balanced per item);
- **agreement** — subject–verb agreement correct vs violated, with is/are
  balanced across both classes (no single token predicts the class);
- **arith_valid** — correct vs off-by-small-delta equations (digit
  distributions overlap across classes).

```text
Hypothesis              : (a) VALIDITY: on token-balanced properties the
                          random-init twin falls to chance — max twin
                          selectivity < 0.05 (mean-pooled AND last-token).
                          (b) SIGNAL: the trained model beats the twin on
                          word_order and agreement — real−twin selectivity
                          > 0.10 on at least 5 layers (FDR-significant).
                          (c) arith_valid: exploratory, no directional
                          prediction at 0.6B.
                          (d) rerun of run #1's core question where
                          accuracy is off ceiling/floor: dataset shift >
                          seed SD on ≥ 1/3 of layers.
Falsification criterion : (a) dies → mean-pooled/last-token reps are
                          unusable even on balanced classes; harness redesign
                          (matched-position tokens) before any atlas map.
                          (b) dies → structural properties are not linearly
                          decodable at 0.6B-4bit: a publishable Level-1
                          negative, and probes/v2 is published as such.
Method                  : capture BOTH mean-pooled and last-token reps in
                          one pass; same grid as run #1 (28 layers × 5 seeds
                          × 2 sets × 3 properties, twin on A sets); word-level
                          class token-overlap certificate stored in the
                          promptset manifest; everything else unchanged.
Baseline / null         : shuffled-label control (every probe); random-init
                          twin; v1 promptsets as positive control (harness
                          must still hit ceiling there — checked on lang_id_A).
Result                  : (pending)
Interpretation          : (pending)
Next experiment         : (pending)
```