spb/modelmap Public License

1---2project: modelmap3document: expA_probe_reliability — hypothesis (registered before run)4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6website: https://modelmap.io7created: 2026-08-128modified: 2026-08-129status: reviewed10---1112# Hypothesis — expA (probe reliability / the noise floor)1314> Probe reliability — noise floor of linear probes across seeds and datasets.15> Registered 2026-08-12, before any run. The harness16> (`src/modelmap/probes/linear.py` + `src/modelmap/stats/replication.py`) is17> implemented and unit-tested against planted structure and pure noise;18> no real-model run has happened yet.1920```text21Hypothesis              : On a 0.5B-class model, per-layer linear-probe maps22                          for simple properties (language ID, code-vs-prose,23                          arithmetic-context) are seed-stable (probe-accuracy24                          SD across ≥5 training seeds < 2 points) but25                          dataset-SENSITIVE: switching prompt corpus moves26                          per-layer accuracy by more than the seed SD on at27                          least a third of layers — i.e. the dataset, not the28                          probe seed, is the dominant noise source29                          (Bolukbasi-consistent).30Falsification criterion : Dies if seed variance ≥ dataset variance across the31                          board (then seeds dominate and every later map needs32                          more seeds, not more corpora); the harness itself is33                          invalid if selectivity on a random-init model34                          exceeds 0.05 (fake structure).35Method                  : Qwen3-0.6B (or Pythia-410M fallback) via mlx-lm;36                          capture residual stream at every layer over ≥237                          disjoint prompt sets per property (versioned,38                          checksummed promptsets); probe_with_control per39                          (layer, property, seed) for 5 seeds; report40                          selectivity, bootstrap CIs, BH-FDR across the layer41                          scan; replication_rate on top-k layer sets.42Baseline / null         : shuffled-label control (built into every probe);43                          random-init model of identical architecture;44                          random-direction projections.45Result                  : (pending)46Interpretation          : (pending — with explicit confidence level)47Next experiment         : (pending — expA output becomes the first map card)48```4950---5152# Hypothesis — expA run #2 (structure-borne properties, token-balanced)5354Registered 2026-08-12 **before** the run, after run #1 failed its validity55gate (twin at ceiling on lexically separable classes). Promptsets v2 are56designed so no token distribution separates the classes:5758- **word_order** — grammatical sentence vs seeded scramble of the SAME words59  (perfectly token-balanced per item);60- **agreement** — subject–verb agreement correct vs violated, with is/are61  balanced across both classes (no single token predicts the class);62- **arith_valid** — correct vs off-by-small-delta equations (digit63  distributions overlap across classes).6465```text66Hypothesis              : (a) VALIDITY: on token-balanced properties the67                          random-init twin falls to chance — max twin68                          selectivity < 0.05 (mean-pooled AND last-token).69                          (b) SIGNAL: the trained model beats the twin on70                          word_order and agreement — real−twin selectivity71                          > 0.10 on at least 5 layers (FDR-significant).72                          (c) arith_valid: exploratory, no directional73                          prediction at 0.6B.74                          (d) rerun of run #1's core question where75                          accuracy is off ceiling/floor: dataset shift >76                          seed SD on ≥ 1/3 of layers.77Falsification criterion : (a) dies → mean-pooled/last-token reps are78                          unusable even on balanced classes; harness redesign79                          (matched-position tokens) before any atlas map.80                          (b) dies → structural properties are not linearly81                          decodable at 0.6B-4bit: a publishable Level-182                          negative, and probes/v2 is published as such.83Method                  : capture BOTH mean-pooled and last-token reps in84                          one pass; same grid as run #1 (28 layers × 5 seeds85                          × 2 sets × 3 properties, twin on A sets); word-level86                          class token-overlap certificate stored in the87                          promptset manifest; everything else unchanged.88Baseline / null         : shuffled-label control (every probe); random-init89                          twin; v1 promptsets as positive control (harness90                          must still hit ceiling there — checked on lang_id_A).91Result                  : (pending)92Interpretation          : (pending)93Next experiment         : (pending)94```95