Confirmatory Tier-0 analysis, paper draft complete for Tier 0, Tier-1 freeze rule, regression test, claim-evidence table

modified README.md +19 −13

				`@@ -14,10 +14,14 @@`
14	> donor language model be transplanted into a recipient that demonstrably lacks it, without full-model	14	> donor language model be transplanted into a recipient that demonstrably lacks it, without full-model
15	> training? This repository holds the preregistered protocol, the experiment harness (toy transformers with	15	> training? This repository holds the preregistered protocol, the experiment harness (toy transformers with
16	> exact hook semantics; MLX/Transformers backends planned), the registry of runs and the paper.	16	> exact hook semantics; MLX/Transformers backends planned), the registry of runs and the paper.
17	−> State on 2026-11-09: Sprints 0–2 done on Tier 0 — donor/recipient gap manufactured and measured	17	+> Result (Tier 0, preregistered confirmatory run, 2026-11-09): a donor-in-the-loop activation graft —
18	−> (donor 1.000 vs recipient 0.092 on the target operator, 3 seeds), hooks verified bitwise no-op, and the	18	+> the recipient's post-layer-0 residual row at every read position replaced by the donor's row through an
19	−> target operator localized by activation patching (executed at the read position by layer-0 attention and	19	+> orthogonal Procrustes map fitted on task-free rows — lifts the recipient from chance (0.06–0.10) to 0.966
20	−> layer-0–2 MLPs; reproducible across seeds, r ≥ 0.96). No transplant has been run yet.	20	+> on held-out Ω programs (control-adjusted effect +0.875, lower bound above 0.25 in 9/9 pairs; control stratum
		21	+> 0.992), carries the rule to unseen operands (0.77 vs 0.94), needs eight of 128 dimensions, survives width,
		22	+> depth and stride-2 mismatches, and is anchored by ablation → rescue (0.11 → 1.00; sham 0.11). Regression-fitted
		23	+> maps (ridge, CCA) annihilate the capability (Proposition 1); donor-free adapters cache; conventional learning
		24	+> on the same items is perfect. Tier 1 (Qwen2.5-1.5B/3B, Llama-3.2-3B) and Tier 2 (Qwen2.5-7B) runs: see status.
21		25
22	---	26	---
23		27
				`@@ -29,7 +33,7 @@`
29	\| SSRN \| not yet submitted \|	33	\| SSRN \| not yet submitted \|
30	\| Journal \| to be assessed after results (ML / mechanistic interpretability venue) \|	34	\| Journal \| to be assessed after results (ML / mechanistic interpretability venue) \|
31	\| Code & data \| synthetic data regenerated from seeds; every run has a manifest in `experiments/manifests/` \|	35	\| Code & data \| synthetic data regenerated from seeds; every run has a manifest in `experiments/manifests/` \|
32	−\| Gates \| A ✔ · B ✔ (Tier 0) · C ✔ (Tier 0) · D–F pending \|	36	+\| Gates \| A ✔ · B ✔ · C ✔ · D ✔ · E ✔ (Tier 0 confirmatory) · F in progress \|
33		37
34	## 🎯 What this paper does	38	## 🎯 What this paper does
35		39
				`@@ -49,15 +53,17 @@ N_D = Q_T(D) − Q_T(D \ S_D) donor necessity`
49		53
50	## 🏆 Headline results	54	## 🏆 Headline results
51		55
52	−No transfer results yet. Pilot (exploratory, run `pilot_tier0_opseq_451ef5087182`):	56	+\| Object \| Estimate (run) \|
53	−
54	−\| Object \| Estimate \|
55	\|---\|---\|	57	\|---\|---\|
56	−\| Donor vs recipient, target stratum (final answer, 3 seeds) \| 1.000 vs 0.092 (chance 0.105); paired gap seed 0 +0.847 [+0.825, +0.870] \|	58	+\| Donor vs recipient, target stratum \| 1.000 vs 0.092; Ω-step 1.000 vs 0.062; control 1.000 both (`pilot_tier0_opseq_451ef5087182`) \|
57	−\| Ω-step accuracy \| donor 1.000, recipient 0.062 \|	59	+\| Localization of Ω in the donor \| residual row at the read position restores 1.00 at every layer; L0 attention 1.00; L0–L2 MLPs 1.00/0.90/0.83; sham 0.00 (pilot) \|
58	−\| Control stratum \| 1.000 vs 1.000 \|	60	+\| Primary graft (confirmatory) \| target 0.966 (min 0.889), control 0.992 (min 0.946); Δ_T^adj +0.875, lower bound > 0.25 in 9/9 pairs → H2 supported; H4 (≥ 0.95 retention everywhere) not supported (`graft_tier0_confirmatory_0c2257f6803e`) \|
59	−\| Hook no-op (6 models) \| bitwise identity in all checks \|	61	+\| Controls (confirmatory) \| sham map 0.091 (control 0.100), norm-matched random 0.091, shuffled item 0.103, shuffled layer 0.186, unrelated-skill donor 0.076 \|
60	−\| Localization of Ω (restored-answer rate) \| 1.00 at the read position for `resid_post` L0–L3 and `attn_out` L0; 0.00 elsewhere; sham 0.00 \|	62	+\| Subspace k = 8 vs random k = 8 \| 0.927 vs 0.077 (+0.850 [+0.837, +0.862]) \|
		63	+\| Rule vs cache \| graft 0.941 seen / 0.768 unseen (ratio 0.82, H5 supported); donor-free adapter 0.702 / 0.000 \|
		64	+\| Ablation → rescue \| 0.113 → 1.000 (sham 0.106) \|
		65	+\| Conventional learning on the same 3 000 items \| supervised adapter 1.000; distillation 1.000 \|
		66	+\| Tier 1 / Tier 2 \| runs in progress (`tier1_taskvector_*`) \|
61		67
62	## 🗂 Repository layout	68	## 🗂 Repository layout
63		69

added docs/CLAIM_EVIDENCE.md +26 −0

++# Claim-to-evidence table — WP-AI-0001 (Gate F)
++
++Every claim in the manuscript maps to a run id, a results key and a generator. "Status" = confirmatory
++(preregistered, new seeds), exploratory (protocol selection), or pilot.
++
++| # | Claim (paper section) | Evidence (run → key) | Generator | Status |
++|---|---|---|---|---|
++| C1 | Donor 1.000 vs recipient 0.092 on the target stratum; Ω-step 1.000 vs 0.062; control 1.000 both (§5.1) | `pilot_tier0_opseq_451ef5087182` → `across_seeds`, `across_seeds_omega_step` | `scripts/run_pilot.py` → `tab_pilot_gap.tex`, `fig_gap_tier0.pdf` | pilot |
++| C2 | Hooks bitwise no-op; MLX wrapper bitwise no-op (§5.2) | pilot → `noop`; `tests/integration/test_mlx_noop.py` | pytest | pilot / test |
++| C3 | Ω executed at the read position: resid patch 1.00 at every layer, attn_out L0 1.00, MLP L0–L2 1.00/0.90/0.83, head 0 of L0 0.62; across-seed r ≥ 0.96 (§5.3) | pilot → `localization` | `scripts/run_pilot.py` → `fig_localization_*.pdf` | pilot |
++| C4 | Primary graft: target 0.966 (sd 0.037, min 0.889), control 0.992 (min 0.946); Δ_T^adj +0.875 (sd 0.042, min +0.776), lower bound > 0.25 in 9/9 pairs → H2 supported; H4 not supported (§5.4) | `graft_tier0_confirmatory_0c2257f6803e` → `CONFIRMATORY.md`, `confirmatory.json` (`primary`) | `scripts/confirmatory_analysis.py`, `tab_confirmatory.tex` | confirmatory |
++| C5 | Controls: sham 0.091 (control 0.100), norm-matched random 0.091, shuffled item 0.103, shuffled layer 0.186 (control 1.000), unrelated donor 0.076 (control 1.000) (§5.4) | confirmatory → `conditions.ctl_*` | `tab_confirmatory.tex` | confirmatory |
++| C6 | Subspace k=8 vs random k=8: +0.850 [+0.837, +0.862] (0.927 vs 0.077) (§5.4) | confirmatory → `secondary.subspace_k8_vs_random`, `holm` | `confirmatory_analysis.py` | confirmatory |
++| C7 | Adapter graft +0.472 [+0.453, +0.490] (0.544); fresh process 0.344 [0.317, 0.372] vs 0.115 (§5.4) | confirmatory → `secondary.adapter_vs_base`, `persistent/fresh_process_eval.json` | `eval_persistent.py` | confirmatory |
++| C8 | Weight graft L0 block vs shuffled: +0.556 [+0.537, +0.577]; retention 0.720 (min 0.310) (§5.4) | confirmatory → `secondary.weight_L0_block_vs_shuffled` | idem | confirmatory |
++| C9 | Rule vs cache: Procrustes seen-fit 0.941 [0.919, 0.960] seen / 0.768 [0.727, 0.808] unseen, ratio 0.816 → H5 supported; adapter 0.702 / 0.000 (§5.4, §6) | confirmatory → `rule_vs_cache` | `fig_mechanism_tier0.pdf` (confirmatory build) | confirmatory |
++| C10 | Ablation 0.113 → rescue 1.000, sham rescue 0.106; reverse graft 0.972 (§5.4, §6) | confirmatory → `mechanism` | idem | confirmatory |
++| C11 | Width 128→96: 0.901 (L0→0), 1.000 (L1→1); depth 4→6: 0.783, 0.999, 1.000, 1.000; stride-2: 0.386, 0.760, 1.000, 1.000; shams 0.08–0.13 (§5.4) | confirmatory → `portability` | `tab_mismatch.tex` (exploratory build), `confirmatory.json` | confirmatory |
++| C12 | Supervised adapter and distillation 1.000 on both strata (§5.4) | confirmatory → `conditions.baseline_*` | `tab_confirmatory.tex` | confirmatory |
++| C13 | Exploratory: identity graft 0.873 same-init / 0.101 diff-init; ridge-on-Ω-rows 0.17; CCA 0.08; ridge task-free 0.81; Procrustes Ω-rows 0.79/0.70; task-free Procrustes 0.993/0.956 (§7, Table main) | `graft_tier0_30e3644aa801` → `conditions`, `across_pairs` | `tab_main_effects.tex`, `fig_pareto_tier0.pdf` | exploratory |
++| C14 | Layer grid 1.00 for donor L1–L3 → any layer, L0→0 0.98, L0→1–3 0.75–0.82; dose 0.44/0.90/0.98/0.92/0.75/0.39 (§7) | v2 → `layer_grid`, `dose` | `fig_layer_dose_tier0.pdf` | exploratory |
++| C15 | Subspace k = 1…32: 0.23, 0.30, 0.65, 0.94, 0.97, 0.98; random 0.08–0.17 (§7) | v2 → `conditions.subspace_*` | `fig_subspace_tier0.pdf` | exploratory |
++| C16 | Steering ≤ 0.10; weight grafts L0 MLP 0.26 (0.96), block 0.64 (0.73), L0–2+embed 0.77 (0.75); Ω embedding only 0.075 (§7) | v2 → `conditions` | `tab_main_effects.tex` | exploratory |
++| C17 | Annihilation proposition (§3) | Appendix A proof | — | theorem |
++| C18 | Tier-1 within-model sufficiency / necessity; cross-model transfer and controls (§5.5) | `tier1_taskvector_*` (dev) and confirmatory split run | `tab_tier1*.tex`, `fig_tier1_transfer.pdf` | pending |
++| C19 | Tier-2 (7B ↔ 3B) | M3U96b run | idem | pending |

modified docs/COMPUTE_BUDGET.md +7 −0

				`@@ -27,4 +27,11 @@`
27	streamed to disk (do not hold in memory).	27	streamed to disk (do not hold in memory).
28	- Tier 2 (7B–14B bf16): 14–28 GB weights; M3U96a/b only; one node per model.	28	- Tier 2 (7B–14B bf16): 14–28 GB weights; M3U96a/b only; one node per model.
29		29
		30	+\| Model matrix (17 toy models × 3 000 steps) \| ≈ 13 min MPS (≈ 45 s each) \|
		31	+\| Transplant run (9 pairs, all families, bootstrap 2 000) \| 140 s (v1), 176 s (v2), 167 s (confirmatory) on MPS \|
		32	+\| Fresh-process persistence check \| ≈ 40 s \|
		33	+\| Tier-1 quick (1 pair, 12 queries/task, bf16 3B + 1.5B) \| 150 s MLX (laptop) \|
		34	+\| Tier-1 dev (3 models localized, 4 pairs, 22–40 queries/task, replace grafts) \| measured in the run manifest (`tier1_taskvector_*`); localization ≈ 10–20 min per model when the GPU is shared \|
		35	+\| Qwen2.5-7B-Instruct bf16 download for Tier 2 (M3U96b) \| ≈ 15 GB \|
		36	+
30	Rule: no parallel heavy runs until the single-node profile is recorded in `EXPERIMENT_LOG.md`.	37	Rule: no parallel heavy runs until the single-node profile is recorded in `EXPERIMENT_LOG.md`.

modified docs/DECISION_LOG.md +4 −0

				`@@ -40,3 +40,7 @@ All entries are dated; "before freeze" refers to the confirmatory freeze (PREREG`
40	method = activation graft with task-free Procrustes at L0→L0, α = 1; confirmatory seed base 90 000. SHA-256 of	40	method = activation graft with task-free Procrustes at L0→L0, α = 1; confirmatory seed base 90 000. SHA-256 of
41	`docs/PREREGISTRATION.md` at freeze recorded below.	41	`docs/PREREGISTRATION.md` at freeze recorded below.
42	PREREGISTRATION.md sha256 at freeze: fe06d326bbbd27916d573a617e94d4443546f179587209e2e04a6893aad25d3e	42	PREREGISTRATION.md sha256 at freeze: fe06d326bbbd27916d573a617e94d4443546f179587209e2e04a6893aad25d3e
		43	+- D11 (2026-11-09, FREEZE Tier 1). Selection rule for the Tier-1/2 confirmatory contrast recorded in
		44	+ PREREGISTRATION before the development results were read (eligibility = own-vector sufficiency ≥ 0.5;
		45	+ selected condition = best dev condition; contrast = selected minus best control on confirmatory queries).
		46	+ PREREGISTRATION.md sha256 at Tier-1 freeze: 85c5db69657144e6d6afecfb4fc346c2345d09455fb6134bee615567e5c705ad

modified docs/PREREGISTRATION.md +19 −1

+ Everything else produced by the script (layer grid, dose, other maps, steering, other k) is reported as
+ exploratory and labelled so in the paper.
++## Tier 1 — selection rule frozen BEFORE the development results are read (2026-11-09)
++
++Development run: `tier1_taskvector_e105e3d91386` (dev split; the selection data). Confirmatory run: same script with
++`--split confirmatory` (disjoint query halves), executed after this rule is recorded and without any change to the
++script's conditions.
++
++- **Eligible tasks** per (pair, task): the recipient's *own* task vector reaches dev zero-shot accuracy ≥ 0.5 at its
++  best layer (within-model sufficiency); otherwise the task is reported as "not vectorizable in the recipient" and
++  excluded from the confirmatory contrast (reported, not tested).
++- **Selected condition** per eligible (pair, task): the transfer condition (map family × recipient layer × α, add
++  variants, rescaled variants, or the donor-in-the-loop replace graft) with the highest dev accuracy; ties broken
++  toward Procrustes, lower layer, α = 1, add before replace.
++- **Confirmatory contrast**: on the confirmatory queries, accuracy of the selected condition minus accuracy of the
++  *best control* on the same queries (max over norm-matched random vector, unrelated-task vector, sham-map vector,
++  mapped donor zero-shot state, shuffled-query donor state), paired on items, bootstrap over items (2 000 draws).
++- **Decision rule (H6, per pair)**: supported if the lower 95 % bound exceeds 0.10 for at least two eligible tasks;
++  "partial" if for one; "not supported" otherwise. Also reported: gain vs recipient zero-shot, recipient few-shot
++  (prompting ceiling), recipient own vector (within-model ceiling), held-out NLL change and cross-task accuracy.
++- Tier 2 (Qwen2.5-7B pairs, run on M3U96b) follows the same rule.

modified docs/REPRODUCIBILITY.md +8 −1

				`@@ -6,7 +6,14 @@ python3 -m pip install -e ".[dev]" # or: uv pip install -e ".[dev]"`
6	python3 -m pytest tests -q # unit + causal-sanity tests (CPU, < 30 s)	6	python3 -m pytest tests -q # unit + causal-sanity tests (CPU, < 30 s)
7	python3 scripts/audit_environment.py # writes docs/ENVIRONMENT_AUDIT.json (add --cluster to probe nodes)	7	python3 scripts/audit_environment.py # writes docs/ENVIRONMENT_AUDIT.json (add --cluster to probe nodes)
8	python3 scripts/run_pilot.py --quick --device cpu --seeds 0 1 --n-boot 200 # CPU smoke test, ≈ 25 s	8	python3 scripts/run_pilot.py --quick --device cpu --seeds 0 1 --n-boot 200 # CPU smoke test, ≈ 25 s
9	−python3 scripts/run_pilot.py --seeds 0 1 2 --steps 3000 # Tier-0 pilot (MPS ≈ minutes)	9	+python3 scripts/run_pilot.py --seeds 0 1 2 --steps 3000 # Tier-0 pilot (MPS ≈ 5 min)
		10	+python3 scripts/train_models.py # 17-model matrix (≈ 13 min MPS)
		11	+python3 scripts/run_transplant.py # exploratory transplant run (≈ 3 min)
		12	+python3 scripts/run_transplant.py --tag confirmatory --seed-base 90000 # confirmatory run (prereg v1.0)
		13	+python3 scripts/eval_persistent.py --run <graft run id> --seed-base 95000 # saved adapter in a fresh process
		14	+python3 scripts/confirmatory_analysis.py --run <confirmatory run id> # pre-declared contrasts + Holm
		15	+python3 scripts/run_tier1.py [--quick] [--split dev\|confirmatory] [--pairs a:b ...] # open-weight models (MLX)
		16	+python3 scripts/build_figures.py --run <run id>; python3 scripts/build_tables.py --graft ... ; scripts/build_paper.sh
10	```	17	```
11		18
12	- Every run writes `experiments/manifests/<run_id>.json` (git commit, config hash, data hashes, seeds,	19	- Every run writes `experiments/manifests/<run_id>.json` (git commit, config hash, data hashes, seeds,

modified docs/TASKS.md +16 −9

				`@@ -25,13 +25,20 @@ Legend: [x] done · [~] in progress · [ ] todo · (gate) quality gate per CLAUD`
25	- [ ] Subspace localization (DAS-style / PCA of Ω-contrast) with norm-matched random subspaces	25	- [ ] Subspace localization (DAS-style / PCA of Ω-contrast) with norm-matched random subspaces
26	- [ ] Necessity test on donor (H3) with control-stratum retention	26	- [ ] Necessity test on donor (H3) with control-stratum retention
27		27
28	−## Sprint 3 — First transplant	28	+## Sprint 3 — First transplant (done 2026-11-09)
29	−- [ ] Alignment module: Procrustes, ridge, CCA, random, unrelated-data maps; leakage audit	29	+- [x] Alignment module: identity, Procrustes, ridge, CCA, random orthogonal; task-row vs task-free fits; leakage audit by item hash
30	−- [ ] Activation graft + sham + norm-matched random + shuffled-token controls	30	+- [x] Activation graft + sham + norm-matched random + shuffled-token/layer + unrelated-donor controls
31	−- [ ] Gate D: reproducibility, overhead disclosure	31	+- [x] Gate D: reproducibility across 9 pairs, overhead disclosed (donor in the loop), map parameter counts
32		32
33	−## Sprint 4–7	33	+## Sprint 4–6 (done 2026-11-09)
34	−- [ ] Tier 1 MLX interventions (layer wrapping) + Transformers equivalence test	34	+- [x] Subspace, steering, adapter, weight, persistent grafts; supervised-adapter and distillation references
35	−- [ ] Portability matrix, persistence (saved adapter), collateral battery	35	+- [x] Layer grid, dose, width/depth mismatch, stride-2; ablation → rescue; reverse graft; rule-vs-cache
36	−- [ ] Confirmatory freeze (PREREGISTRATION v1.0) and run	36	+- [x] Confirmatory freeze (PREREGISTRATION v1.0, D10) and confirmatory run + analysis (H2, H5 supported; H4 not)
37	−- [ ] Paper, figures, claim-evidence table (Gate F)	37	+- [x] Tier-1 MLX layer wrapping with bitwise no-op test; task-vector localization + transfer script
		38	+- [~] Tier-1 development run (4 pairs) → Tier-1 freeze → confirmatory split run
		39	+- [~] Tier-2: Qwen2.5-7B ↔ 3B pairs on M3U96b
		40	+- [ ] Transformers equivalence test for the MLX backend (not done; reported as a limitation)
		41	+
		42	+## Sprint 7 — Manuscript
		43	+- [~] Paper sections written for Tier 0; Tier 1 pending; claim-evidence table (Gate F) pending
		44	+- [ ] Internal multi-agent review (UQO-WP-REVIEW pipeline)

modified paper/appendix/appendix.tex +12 −4

				`@@ -32,8 +32,8 @@ tested; the experiments in \S\ref{sec:results} estimate how far from it the fitt`
32		32
33	\section{Configurations, splits and model cards}\label{app:configs}	33	\section{Configurations, splits and model cards}\label{app:configs}
34	Tables of split hashes, model specifications and run identifiers are generated from the run manifests	34	Tables of split hashes, model specifications and run identifiers are generated from the run manifests
35	−(\texttt{experiments/manifests/}) by \texttt{scripts/build\_tables.py}; the registry	35	+(\texttt{experiments/manifests/}) by \url{scripts/build_tables.py}; the registry
36	−(\texttt{experiments/registry/runs.jsonl}) lists every planned, running, failed and completed run.	36	+(\url{experiments/registry/runs.jsonl}) lists every planned, running, failed and completed run.
37		37
38	\begin{table}[h]\centering\scriptsize	38	\begin{table}[h]\centering\scriptsize
39	\caption{Tier-0 model matrix (\texttt{artifacts/models/matrix\_v1/index.json}): architecture, excluded and	39	\caption{Tier-0 model matrix (\texttt{artifacts/models/matrix\_v1/index.json}): architecture, excluded and
				`@@ -57,8 +57,16 @@ same- and different-initialization pairs. High $R^2$ does not predict transfer (`
57	\end{table}	57	\end{table}
58		58
59	\section{Tier-1 prompts and tasks}\label{app:tier1}	59	\section{Tier-1 prompts and tasks}\label{app:tier1}
60	−Few-shot prompt: $k{=}8$ blocks ``\texttt{Input: <x>\textbackslash nOutput: <y>\textbackslash n\textbackslash n}''	60	+Few-shot prompt: $k{=}8$ demonstration blocks followed by the query block; zero-shot prompt: the query block only.
61	−followed by ``\texttt{Input: <query>\textbackslash nOutput:}''; zero-shot prompt: the last block only.	61	+\begin{verbatim}
		62	+Input: <x_1>
		63	+Output: <y_1>
		64	+
		65	+...
		66	+
		67	+Input: <query>
		68	+Output:
		69	+\end{verbatim}
62	Alignment prompts: the same frame with 0--3 random demonstration pairs whose outputs are random and a random	70	Alignment prompts: the same frame with 0--3 random demonstration pairs whose outputs are random and a random
63	query (sentences, words, numbers, names), none of which encodes a task. Task item lists are in	71	query (sentences, words, numbers, names), none of which encodes a task. Task item lists are in
64	\texttt{src/wp\_ai\_0001/datasets/tier1\_tasks.py} with their hashes in the run manifests.	72	\texttt{src/wp\_ai\_0001/datasets/tier1\_tasks.py} with their hashes in the run manifests.

modified paper/main.pdf +0 −0

Binary file not shown.

modified paper/main.tex +2 −0

				`@@ -42,6 +42,8 @@`
42		42
43	% --- Figures ---	43	% --- Figures ---
44	\usepackage{graphicx}	44	\usepackage{graphicx}
		45	+\usepackage{url}
		46	+\urlstyle{tt}
45	\usepackage{subcaption}	47	\usepackage{subcaption}
46	\usepackage{float}	48	\usepackage{float}
47		49

modified paper/sections/experimental_design.tex +1 −1

				`@@ -13,7 +13,7 @@ recomputation, hashed, and checked for item-level disjointness against all train`
13	evaluation splits involved in a run; split hashes are stored in run manifests. Design decision D2 (trace	13	evaluation splits involved in a run; split hashes are stored in run manifests. Design decision D2 (trace
14	instead of a single answer token) was taken after a single-answer variant memorized without generalizing;	14	instead of a single answer token) was taken after a single-answer variant memorized without generalizing;
15	its consequence---that the Tier-0 capability is a per-step map once the trace is visible---is controlled	15	its consequence---that the Tier-0 capability is a per-step map once the trace is visible---is controlled
16	−by the \texttt{trace\_stride} knob (stride 2 forces two operators per emitted result).	16	+by the \url{trace_stride} knob (stride 2 forces two operators per emitted result).
17		17
18	\paragraph{Models and roles.} Decoder-only transformers (4 layers, $d{=}128$, 4 heads, $0.80$\,M parameters;	18	\paragraph{Models and roles.} Decoder-only transformers (4 layers, $d{=}128$, 4 heads, $0.80$\,M parameters;
19	pre-LN, explicit attention with per-head outputs exposed) are trained from scratch with teacher forcing on	19	pre-LN, explicit attention with per-head outputs exposed) are trained from scratch with teacher forcing on

modified paper/sections/limitations.tex +2 −2

				`@@ -36,6 +36,6 @@ detection tools for such transfers.`
36		36
37	\paragraph{Reproducibility.} Every run has a manifest with git commit, configuration hash, split hashes, seeds,	37	\paragraph{Reproducibility.} Every run has a manifest with git commit, configuration hash, split hashes, seeds,
38	library versions and hardware; the registry lists every planned, running, failed and completed run; figures and	38	library versions and hardware; the registry lists every planned, running, failed and completed run; figures and
39	−tables are regenerated from results files by \texttt{scripts/build\_figures.py} and \texttt{scripts/build\_tables.py};	39	+tables are regenerated from results files by \url{scripts/build_figures.py} and \url{scripts/build_tables.py};
40	−\texttt{docs/REPRODUCIBILITY.md} gives the commands (CPU smoke test $\approx$ 25\,s; Tier-0 pilot $\approx$ 5\,min;	40	+\url{docs/REPRODUCIBILITY.md} gives the commands (CPU smoke test $\approx$ 25\,s; Tier-0 pilot $\approx$ 5\,min;
41	Tier-0 transplant run $\approx$ 3\,min on an M5 Max; Tier-1 run minutes to an hour depending on pairs).	41	Tier-0 transplant run $\approx$ 3\,min on an M5 Max; Tier-1 run minutes to an hour depending on pairs).

modified paper/sections/operational_theory.tex +2 −2

				`@@ -39,7 +39,7 @@ We implement six families (Table~\ref{tab:families}).`
39	\caption{Transplant families, their inference-time dependency on the donor, what is fitted, and the matched	39	\caption{Transplant families, their inference-time dependency on the donor, what is fitted, and the matched
40	control used for the control-adjusted effect.}	40	control used for the control-adjusted effect.}
41	\label{tab:families}	41	\label{tab:families}
42	−\begin{tabular}{llll}	42	+\resizebox{\textwidth}{!}{\begin{tabular}{llll}
43	\toprule	43	\toprule
44	Family & donor at inference & fitted object (data) & matched control \\	44	Family & donor at inference & fitted object (data) & matched control \\
45	\midrule	45	\midrule
				`@@ -50,7 +50,7 @@ Adapter graft & no & affine $f: x_R \to A(x_D)$ (ridge) & supervised adapter wit`
50	Weight graft & no & none (copy of blocks / rows) & shuffled-layer / unrelated-donor weights \\	50	Weight graft & no & none (copy of blocks / rows) & shuffled-layer / unrelated-donor weights \\
51	Persistent graft & no & saved adapter, fresh process & unmodified recipient on new items \\	51	Persistent graft & no & saved adapter, fresh process & unmodified recipient on new items \\
52	\bottomrule	52	\bottomrule
53	−\end{tabular}	53	+\end{tabular}}
54	\end{table} The \emph{activation graft} replaces the recipient's	54	\end{table} The \emph{activation graft} replaces the recipient's
55	residual row at a read position by a mapped donor row computed on the same token ids	55	residual row at a read position by a mapped donor row computed on the same token ids
56	($x_R \leftarrow x_R + \alpha(A(x_D) - x_R)$); the \emph{subspace graft} restricts the replacement to a	56	($x_R \leftarrow x_R + \alpha(A(x_D) - x_R)$); the \emph{subspace graft} restricts the replacement to a

modified paper/sections/results.tex +12 −10

				`@@ -1,12 +1,14 @@`
1	\section{Results}\label{sec:results}	1	\section{Results}\label{sec:results}
2		2
3	−\paragraph{Provenance.} Every number below names its run: the pilot \texttt{pilot\_tier0\_opseq\_451ef5087182}	3	+\begin{sloppypar}
4	−(Gates B--C and localization), the exploratory transplant runs \texttt{graft\_tier0\_f50a28ca1bf7} (protocol v1)	4	+\paragraph{Provenance.} Every number below names its run: the pilot \url{pilot_tier0_opseq_451ef5087182}
5	−and \texttt{graft\_tier0\_30e3644aa801} (v2), the confirmatory run	5	+(Gates B--C and localization), the exploratory transplant runs \url{graft_tier0_f50a28ca1bf7} (protocol v1)
6	−\texttt{graft\_tier0\_confirmatory\_0c2257f6803e} (preregistration v1.0, new seeds), and the Tier-1 runs of	6	+and \url{graft_tier0_30e3644aa801} (v2), the confirmatory run
		7	+\url{graft_tier0_confirmatory_0c2257f6803e} (preregistration v1.0, new seeds), and the Tier-1 runs of
7	\S\ref{sec:tier1}. Tables and figures are regenerated from the results files by	8	\S\ref{sec:tier1}. Tables and figures are regenerated from the results files by
8	−\texttt{scripts/build\_tables.py} and \texttt{scripts/build\_figures.py}; the confirmatory decision rules are	9	+\url{scripts/build_tables.py} and \url{scripts/build_figures.py}; the confirmatory decision rules are
9	−evaluated by \texttt{scripts/confirmatory\_analysis.py}.	10	+evaluated by \url{scripts/confirmatory_analysis.py}.
		11	+\end{sloppypar}
10		12
11	\subsection{Gate B: a manufactured, reproducible capability gap}	13	\subsection{Gate B: a manufactured, reproducible capability gap}
12	Three donor seeds trained on all eight operators and three recipient seeds trained with $\Omega$ excluded	14	Three donor seeds trained on all eight operators and three recipient seeds trained with $\Omega$ excluded
				`@@ -19,7 +21,7 @@ roles fail the length-OOD stratum ($0.086$ vs $0.091$), a limitation of absolute`
19	treat as a stratum, not a target.	21	treat as a stratum, not a target.
20		22
21	\begin{table}[t]\centering\small	23	\begin{table}[t]\centering\small
22	−\caption{Pilot gap qualification (run \texttt{pilot\_tier0\_opseq\_451ef5087182}): final-answer exact match by	24	+\caption{Pilot gap qualification (run \url{pilot_tier0_opseq_451ef5087182}): final-answer exact match by
23	stratum, mean (sd) over three seeds per role; paired gap with cluster-bootstrap 95\% interval over programs.}	25	stratum, mean (sd) over three seeds per role; paired gap with cluster-bootstrap 95\% interval over programs.}
24	\label{tab:pilot}	26	\label{tab:pilot}
25	\input{tables/tab_pilot_gap}	27	\input{tables/tab_pilot_gap}
				`@@ -36,7 +38,7 @@ majority-class chance level (grey dashes).}`
36	For all six pilot models, logits computed with (i) no hooks, (ii) capture hooks on all 27 hook points, and	38	For all six pilot models, logits computed with (i) no hooks, (ii) capture hooks on all 27 hook points, and
37	(iii) a modifying hook registered but disabled are bitwise identical on 256 items, and repeated runs are	39	(iii) a modifying hook registered but disabled are bitwise identical on 256 items, and repeated runs are
38	bitwise identical on MPS; the same modifying hook, when enabled, changes the output. The MLX backend passes	40	bitwise identical on MPS; the same modifying hook, when enabled, changes the output. The MLX backend passes
39	−the same test on Qwen2.5-1.5B and Llama-3.2-3B (\texttt{tests/integration/test\_mlx\_noop.py}).	41	+the same test on Qwen2.5-1.5B and Llama-3.2-3B (\url{tests/integration/test_mlx_noop.py}).
40		42
41	\subsection{RQ1: where the donor executes $\Omega$}	43	\subsection{RQ1: where the donor executes $\Omega$}
42	On 490 clean/corrupted pairs the clean run scores $1.000$ and the sham (corrupted run scored on the clean	44	On 490 clean/corrupted pairs the clean run scores $1.000$ and the sham (corrupted run scored on the clean
				`@@ -73,7 +75,7 @@ destroys the control stratum ($0.100$); norm-matched random rows and shuffled-it`
73	shuffled-layer rows ($0.186$) and the unrelated-skill donor ($0.076$) leave the control stratum intact.	75	shuffled-layer rows ($0.186$) and the unrelated-skill donor ($0.076$) leave the control stratum intact.
74		76
75	\begin{table}[t]\centering\scriptsize	77	\begin{table}[t]\centering\scriptsize
76	−\caption{Confirmatory run \texttt{graft\_tier0\_confirmatory\_0c2257f6803e}: target- and control-stratum	78	+\caption{Confirmatory run \url{graft_tier0_confirmatory_0c2257f6803e}: target- and control-stratum
77	accuracy by condition, mean (sd) over same-initialization and different-initialization pairs, and the	79	accuracy by condition, mean (sd) over same-initialization and different-initialization pairs, and the
78	control-adjusted effect against the sham map (mean of per-pair cluster-bootstrap estimates and bounds).	80	control-adjusted effect against the sham map (mean of per-pair cluster-bootstrap estimates and bounds).
79	Families other than the activation graft are evaluated on the three matched-seed pairs.}	81	Families other than the activation graft are evaluated on the three matched-seed pairs.}
				`@@ -142,5 +144,5 @@ $\Omega$ rows with operands in $\{0,\dots,5\}$, evaluated on items whose operand`
142	\label{fig:mechanism}\end{figure}	144	\label{fig:mechanism}\end{figure}
143		145
144	\subsection{Tier 1: open-weight models}\label{sec:tier1}	146	\subsection{Tier 1: open-weight models}\label{sec:tier1}
145	−[PLACEHOLDER --- filled from runs \texttt{tier1\_taskvector\_e105e3d91386} (development) and the confirmatory	147	+[PLACEHOLDER --- filled from runs \url{tier1_taskvector_e105e3d91386} (development) and the confirmatory
146	Tier-1 run after the Tier-1 freeze; see \S\ref{sec:design}.]	148	Tier-1 run after the Tier-1 freeze; see \S\ref{sec:design}.]

modified paper/sections/robustness.tex +3 −3

				`@@ -1,11 +1,11 @@`
1	\section{Robustness and Failure Modes}\label{sec:robustness}	1	\section{Robustness and Failure Modes}\label{sec:robustness}
2		2
3	−All numbers in this section are \emph{exploratory} (protocol v2 run \texttt{graft\_tier0\_30e3644aa801},	3	+All numbers in this section are \emph{exploratory} (protocol v2 run \url{graft_tier0_30e3644aa801},
4	matched seed-0 pair unless stated); they motivated the frozen protocol and are not re-tested on the	4	matched seed-0 pair unless stated); they motivated the frozen protocol and are not re-tested on the
5	confirmatory items except where \S\ref{sec:results} says so.	5	confirmatory items except where \S\ref{sec:results} says so.
6		6
7	\begin{table}[t]\centering\scriptsize	7	\begin{table}[t]\centering\scriptsize
8	−\caption{Exploratory run \texttt{graft\_tier0\_30e3644aa801} (protocol v2): target- and control-stratum	8	+\caption{Exploratory run \url{graft_tier0_30e3644aa801} (protocol v2): target- and control-stratum
9	accuracy by condition, mean (sd) over same-initialization and different-initialization pairs, and the	9	accuracy by condition, mean (sd) over same-initialization and different-initialization pairs, and the
10	control-adjusted effect against the sham map.}	10	control-adjusted effect against the sham map.}
11	\label{tab:main}	11	\label{tab:main}
				`@@ -14,7 +14,7 @@ control-adjusted effect against the sham map.}`
14		14
15	\begin{table}[t]\centering\scriptsize	15	\begin{table}[t]\centering\scriptsize
16	\caption{Architectural mismatch and stride-2 transfer (exploratory run): target accuracy, $\Omega$-step	16	\caption{Architectural mismatch and stride-2 transfer (exploratory run): target accuracy, $\Omega$-step
17	−accuracy, control accuracy and in-sample $R^2$ of the map; \texttt{unrelated\_procrustes} = Procrustes fitted	17	+accuracy, control accuracy and in-sample $R^2$ of the map; \url{unrelated_procrustes} = Procrustes fitted
18	on task-free rows; \texttt{sham} = random orthogonal map.}	18	on task-free rows; \texttt{sham} = random orthogonal map.}
19	\label{tab:mismatch}	19	\label{tab:mismatch}
20	\resizebox{\textwidth}{!}{\input{tables/tab_mismatch}}	20	\resizebox{\textwidth}{!}{\input{tables/tab_mismatch}}

modified scripts/run_tier1.py +1 −0

				`@@ -40,6 +40,7 @@ MODELS = {`
40	"qwen1.5b": ("mlx-community/Qwen2.5-1.5B-Instruct-bf16", "Qwen2.5", "Apache-2.0"),	40	"qwen1.5b": ("mlx-community/Qwen2.5-1.5B-Instruct-bf16", "Qwen2.5", "Apache-2.0"),
41	"qwen3b": ("mlx-community/Qwen2.5-3B-Instruct-bf16", "Qwen2.5", "Qwen Research License"),	41	"qwen3b": ("mlx-community/Qwen2.5-3B-Instruct-bf16", "Qwen2.5", "Qwen Research License"),
42	"llama3b": ("mlx-community/Llama-3.2-3B-Instruct-bf16", "Llama-3.2", "Llama 3.2 Community License"),	42	"llama3b": ("mlx-community/Llama-3.2-3B-Instruct-bf16", "Llama-3.2", "Llama 3.2 Community License"),
		43	+ "qwen7b": ("mlx-community/Qwen2.5-7B-Instruct-bf16", "Qwen2.5", "Apache-2.0"), # Tier 2 (run on a 96 GB node)
43	}	44	}
44		45
45		46

modified src/wp_ai_0001/models/__init__.py +7 −2

				`@@ -1,2 +1,7 @@`
1	−from wp_ai_0001.models.hooks import HookManager, Intervention # noqa: F401	1	+"""Model adapters. PyTorch toy models are imported lazily so that the MLX backend can be used on hosts
2	−from wp_ai_0001.models.toy_transformer import ToyTransformer # noqa: F401	2	+without torch (e.g. MLX-only cluster nodes)."""
		3	+try: # pragma: no cover - optional dependency
		4	+ from wp_ai_0001.models.hooks import HookManager, Intervention # noqa: F401
		5	+ from wp_ai_0001.models.toy_transformer import ToyTransformer # noqa: F401
		6	+except ModuleNotFoundError: # torch absent
		7	+ pass

added tests/regression/test_confirmatory_numbers.py +28 −0

++"""Regression: the numbers quoted in the paper for the confirmatory run are those in the registered results."""
++import json
++from pathlib import Path
++
++import pytest
++
++ROOT = Path(__file__).resolve().parents[2]
++RUN = "graft_tier0_confirmatory_0c2257f6803e"
++P = ROOT / "experiments" / "summaries" / RUN / "confirmatory.json"
++
++
++@pytest.mark.skipif(not P.exists(), reason="confirmatory results not present")
++def test_primary_contrast_as_reported():
++    c = json.loads(P.read_text())
++    prim = c["primary"]
++    assert prim["n_pairs_lower_bound_above_sesoi"] == 9
++    assert abs(prim["mean_adj"] - 0.875) < 0.0015
++    assert prim["H2_supported"] is True and prim["H4_retention_supported"] is False
++    assert abs(c["secondary"]["subspace_k8_vs_random"]["estimate"] - 0.850) < 0.0015
++    assert abs(c["secondary"]["rule_vs_cache"]["ratio"] - 0.816) < 0.0015
++    assert c["mechanism"]["mech_rescue_from_intact_donor"]["donor_s1->donor_s0(ablated)"] == 1.0
++
++
++@pytest.mark.skipif(not P.exists(), reason="confirmatory results not present")
++def test_manifest_completed_and_hashes_recorded():
++    m = json.loads((ROOT / "experiments" / "manifests" / f"{RUN}.json").read_text())
++    assert m["status"] == "completed"
++    assert {"target_iid", "control_iid", "align_target", "align_unrelated"} <= set(m["data_hashes"])

spb/wpai0001_uqo

Theme