# Experiment plan — WP-AI-0001

## 1. Tiers and model matrix

| Tier | Models | Purpose | Backend | Status |
|---|---|---|---|---|
| 0 | toy decoder transformers trained from scratch (0.8 M params default; 4 layers, d = 128) | full visibility, controlled gap, exhaustive interventions | PyTorch (CPU/MPS) | **pilot running** |
| 1 | Qwen2.5-1.5B-Instruct, Qwen2.5-3B-Instruct, Llama-3.2-3B-Instruct (bf16, cached locally) | external validity, same-family different-size, cross-family, cross-tokenizer | MLX (layer-wrapping interventions) with a PyTorch/Transformers equivalence check | planned (after Gate D on Tier 0) |
| 2 | 7B–14B open weights (e.g. Qwen2.5-7B/14B, Llama-3.1-8B) | scale | MLX on M3U96a/b (96 GB) | conditional |
| 3 | larger | only with explicit approval | — | not planned |

Pairs (Tier 1): Qwen2.5-1.5B→3B (same family, same tokenizer, 28→36 layers); Llama-3.2-3B↔Qwen2.5-3B
(different family, different tokenizer, 28 vs 36 layers); Tier 0: same architecture / different seeds,
same architecture / different widths, different depths.

## 2. Task families

- **A — OPSEQ** (implemented): invented operators on Z_11, scratchpad trace; strata `target_iid`,
  `control_iid`, `comp_heldout`, `len_ood`; difficulty knob `trace_stride`.
- **B — FSA** (generator implemented): random DFA execution over an invented alphabet.
- **C — MORPH** (generator implemented): context-dependent suffix rule on pseudo-words; Tier 1 variant
  with natural-language pseudo-morphology to be specified at Sprint 4.
- Negative-control battery (Tier 1): unrelated arithmetic, factual recall, instruction following,
  held-out text perplexity, calibration, refusal behaviour.

## 3. Phases (per CLAUDE.md §6) and their scripts

| Phase | Script | Output |
|---|---|---|
| 1 Baseline & gap | `scripts/run_pilot.py` | gap table, Ω-step accuracies, no-op checks |
| 2 Localization | `scripts/run_pilot.py` (within-donor patching, ablation, heads) | heatmaps, reproducibility |
| 3 Alignment | `scripts/run_alignment.py` (to write) | Procrustes / ridge / CCA / random / unrelated-data maps with leakage audit |
| 4 Transplant | `scripts/run_transplant.py` (to write) | activation graft, subspace graft, weight graft (same-arch), adapter graft |
| 5 Controls | same | sham, norm-matched random, unrelated-skill donor, shuffled token/layer, few-shot, steering, matched adapter FT, distillation |
| 6 Robustness | `scripts/run_sweeps.py` (to write) | strength × layer × position × dimension × pair |
| 7 Mechanism | `scripts/run_mechanism.py` (to write) | ablation→loss→rescue, bidirectional, dose-response |
| 8 Confirmatory | `scripts/run_confirmatory.py` (to write) | frozen protocol on untouched splits |

## 4. Minimal viable experiment (MVE) and go/no-go

MVE = Tier 0, OPSEQ, Ω = `qua`, donor (all operators) vs recipient (Ω excluded), 3 seeds each.
Go to Sprint 3 (first graft) iff: (a) donor Ω-step accuracy ≥ 0.95 and recipient ≤ chance + 0.1 on
`target_iid`; (b) all no-op checks true; (c) at least one localization cell with restored-answer rate
≥ 0.8 and across-seed heatmap correlation ≥ 0.8. Otherwise diagnose (see RISK_REGISTER R1–R3).

## 5. Pre-declared confirmatory contrasts (Tier 0)

1. activation graft vs matched sham (primary); 2. subspace graft vs norm-matched random subspace;
3. weight graft (same architecture) vs shuffled-layer weight graft; 4. adapter graft vs matched-size
adapter trained on the same alignment data without donor signal. Each: Δ_T^adj with cluster-bootstrap CI,
Holm-corrected across the four families; retention on `control_iid`.

## 6. Figures that the plan can support (data-dependent)

Gap distributions (F2), localization heatmap (F3), Pareto target-vs-retention (F5), portability matrix
(F6), graft size vs effect (F7), ablation→rescue chain (F8), robustness (F9), failure taxonomy (F10).