SPB Git
spboucher.ai

spb/wpai0001_uqo

Public

Tier-1/2 runs (dev + confirmatory, amendments A1-A2), frozen Tier-1 analysis, paper v1.0 draft (Tier-2 confirmatory paragraph pending)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Nov 9, 2026) parent 043b146 Browse files

14 changed files +19,967 −3

14 changed files
Load all 14 files Unified Split
modified docs/DECISION_LOG.md +4 −0
@@ -44,3 +44,7 @@ All entries are dated; "before freeze" refers to the confirmatory freeze (PREREG
44 PREREGISTRATION before the development results were read (eligibility = own-vector sufficiency ≥ 0.5; 44 PREREGISTRATION before the development results were read (eligibility = own-vector sufficiency ≥ 0.5;
45 selected condition = best dev condition; contrast = selected minus best control on confirmatory queries). 45 selected condition = best dev condition; contrast = selected minus best control on confirmatory queries).
46 PREREGISTRATION.md sha256 at Tier-1 freeze: 85c5db69657144e6d6afecfb4fc346c2345d09455fb6134bee615567e5c705ad 46 PREREGISTRATION.md sha256 at Tier-1 freeze: 85c5db69657144e6d6afecfb4fc346c2345d09455fb6134bee615567e5c705ad
47+- **D12 (2026-11-09, amendment A1 before the Tier-1 confirmatory run).** Shuffled-query donor state reclassified
48+ from null control to mechanism diagnostic (it carries the task identity); both analyses reported.
49+- **D13 (2026-11-09, amendment A2).** Tier-1/2 confirmatory runs load the development localization (layers,
50+ vectors) instead of re-localizing on confirmatory queries; first confirmatory run invalidated and re-run.
modified docs/EXPERIMENT_LOG.md +4 −0
@@ -83,3 +83,7 @@ mechanism unchanged (ablation 0.086 → rescue 1.000, sham 0.115; reverse 0.965)
83 83
84 | 2026-11-09 | `graft_tier0_confirmatory_0c2257f6803e` (MPS, PREREGISTRATION v1.0, seed base 90 000) | **completed** | `experiments/summaries/graft_tier0_confirmatory_0c2257f6803e/{results.json,CONFIRMATORY.md,confirmatory.json}` | primary: 9/9 pairs lower bound > SESOI, mean Δ_T^adj 0.875 (sd 0.042, min 0.776) → H2 SUPPORTED; H4 NOT SUPPORTED (one pair control 0.946 < 0.95; others ≥ 0.985); Holm: subspace k8 vs random +0.850, adapter vs base +0.472, weight L0 block vs shuffled +0.556 (retention 0.720), all rejected at Holm; H5 seen 0.941 / unseen 0.768 (ratio 0.816 ≥ 0.8) SUPPORTED; adapter cache 0.702 / 0.000 | 84 | 2026-11-09 | `graft_tier0_confirmatory_0c2257f6803e` (MPS, PREREGISTRATION v1.0, seed base 90 000) | **completed** | `experiments/summaries/graft_tier0_confirmatory_0c2257f6803e/{results.json,CONFIRMATORY.md,confirmatory.json}` | primary: 9/9 pairs lower bound > SESOI, mean Δ_T^adj 0.875 (sd 0.042, min 0.776) → H2 SUPPORTED; H4 NOT SUPPORTED (one pair control 0.946 < 0.95; others ≥ 0.985); Holm: subspace k8 vs random +0.850, adapter vs base +0.472, weight L0 block vs shuffled +0.556 (retention 0.720), all rejected at Holm; H5 seen 0.941 / unseen 0.768 (ratio 0.816 ≥ 0.8) SUPPORTED; adapter cache 0.702 / 0.000 |
85 | 2026-11-09 | `eval_persistent` on the confirmatory run (fresh process, seed base 95 000) | completed | `.../persistent/fresh_process_eval.json` | saved adapter: target 0.344 [0.317, 0.372] vs base 0.115; control 0.979 | 85 | 2026-11-09 | `eval_persistent` on the confirmatory run (fresh process, seed base 95 000) | completed | `.../persistent/fresh_process_eval.json` | saved adapter: target 0.344 [0.317, 0.372] vs base 0.115; control 0.979 |
86+| 2026-11-09 | `tier1_taskvector_e105e3d91386` (MLX laptop, dev split, 4 pairs, k = 8, 10 vector sets) | completed (1 649 s) | `experiments/summaries/tier1_taskvector_e105e3d91386/results.json` | within-model sufficiency: antonym 0.84–0.88 (L14–L25), capital 0.82–1.00 (L14–L25), addthree only in Llama (0.80, L14), morph none; cross-model best (dev): antonym 0.66–0.81 in all 4 pairs vs best null control 0.38–0.56; capital 0.73 (1.5B→3B, Procrustes L28), 0.50 (Llama→Qwen3B, CCA L18), 0.18 (3B→1.5B), 0.05 (3B→Llama); shuffled-query donor state transfers antonym as well as the matched query (→ amendment A1, D12) |
87+| 2026-11-09 | `tier1_taskvector_eb5d091bf5ed` (MLX on M3U96b, Tier 2: Qwen2.5-7B ↔ Qwen2.5-3B, Qwen2.5-7B → Llama-3.2-3B, dev split) | completed (2 528 s) | `experiments/summaries/tier1_taskvector_eb5d091bf5ed/results.json` (pulled from the node) | Qwen-7B within-model: antonym 0.91 (L16), capital 1.00 (L11), morph 0.03, add-three 0.00; dev transfer: antonym 0.84 both Qwen directions (controls 0.53/0.59), capital 0.91 (3B→7B; unrelated-task control 0.77) / 0.14 (7B→3B); 7B→Llama antonym 0.38 vs control 0.34 (no transfer) |
88+| 2026-11-09 | `tier1_taskvector_confirmatory_82a78041c0ce` | **invalidated** | manifest | re-localized on confirmatory queries (A2/D13); no number used |
89+| 2026-11-09 | `tier1_taskvector_confirmatory_625c54bd186f` (laptop, confirmatory split, dev-frozen localization) | completed | `experiments/summaries/tier1_taskvector_confirmatory_625c54bd186f/{results.json,TIER1_CONFIRMATORY.md}` | H6 per pair (frozen rule, amended controls): Qwen3B→1.5B not supported (antonym +0.16 [0.00, 0.34], capital +0.09); 1.5B→3B partial (capital +0.36 [0.14, 0.59] ✔, antonym +0.28 [0.09, 0.47]); Llama→Qwen3B partial (antonym +0.50 [0.31, 0.69] ✔, capital +0.18 [0.05, 0.36]); Qwen3B→Llama not supported (antonym +0.25 [0.09, 0.41], capital −0.05); add-three/morph not eligible or 0 |
modified docs/PREREGISTRATION.md +16 −0
@@ -130,3 +130,19 @@ script's conditions.
130 "partial" if for one; "not supported" otherwise. Also reported: gain vs recipient zero-shot, recipient few-shot 130 "partial" if for one; "not supported" otherwise. Also reported: gain vs recipient zero-shot, recipient few-shot
131 (prompting ceiling), recipient own vector (within-model ceiling), held-out NLL change and cross-task accuracy. 131 (prompting ceiling), recipient own vector (within-model ceiling), held-out NLL change and cross-task accuracy.
132 - Tier 2 (Qwen2.5-7B pairs, run on M3U96b) follows the same rule. 132 - Tier 2 (Qwen2.5-7B pairs, run on M3U96b) follows the same rule.
133+
134+### Amendment A1 to the Tier-1 rule (2026-11-09, recorded BEFORE the Tier-1 confirmatory run; D12)
135+The first development pair showed that the "mapped donor few-shot state of a *shuffled* query" condition is not a
136+null control: the recipient keeps its own query in context, and the donor state carries the task identity, so it
137+transfers the task exactly like the matched-query state (antonym, qwen3b→qwen1.5b: 0.72 vs 0.72 on dev). It is
138+reclassified as a **mechanism diagnostic** (task identity vs query content). The null-control set for the H6
139+contrast is therefore: norm-matched random vector, unrelated-task vector, sham-map vector (random orthogonal map),
140+and mapped donor **zero-shot** state (same map, no task information). The analysis reports both the pre-amendment
141+(shuffled query included) and the amended contrast.
142+
143+### Amendment A2 (2026-11-09, procedural; D13)
144+The first confirmatory Tier-1 run (`tier1_taskvector_confirmatory_82a78041c0ce`) re-localized the recipient's
145+best layer on the confirmatory queries, so some development-selected conditions did not exist in it. As the rule
146+intended, localization (layers, vectors) is frozen from the development split; the run is marked *invalidated*
147+in the registry and re-executed with `--split confirmatory` loading the development localization caches. No
148+numbers from the invalidated run are used.
added docs/review_enrolment/_constraints_WP-AI-0001.md +8 −0
@@ -0,0 +1,8 @@
1+## Paper-specific constraints (binding for every stage of this review)
2+- **Enrolment.** WP-AI-0001 was built autonomously on 2026-11-09 (sprints 0–7 in one session) and enrolled the same day for the full pipeline (Stages A–G) under the standing instructions of the earlier waves: Stages C–G run autonomously; every authorization request is treated as authorised EXCEPT paid API spend (0 USD); the review branch is pushed after every commit; `main` is never touched; one paper at a time; ≤ 7 concurrent subagents.
3+- **Genre.** Mechanistic-interpretability / machine-learning methods paper with a preregistered confirmatory test: can a causally localized subsystem ("organ", a metaphor) be transplanted from a donor language model into a recipient that lacks the capability, without training? Tier 0 = toy decoder transformers (0.80 M params, 4 layers) trained from scratch on an invented operator micro-world (OPSEQ, scratchpad trace, modulus 11), donor = all 8 operators, recipient = operator Ω excluded; localization by interchange interventions; six transplant families (activation, subspace, steering, adapter, weight, persistent) against a control battery (sham map, norm-matched random, shuffled item/layer, unrelated-skill donor) and conventional-learning references (supervised adapter, distillation); PREREGISTRATION v1.0 frozen before the confirmatory run (`graft_tier0_confirmatory_0c2257f6803e`, seed base 90 000); Tier 1 = Qwen2.5-1.5B/3B-Instruct and Llama-3.2-3B-Instruct in bf16 under MLX, task vectors (few-shot − zero-shot last-token state) localized within model and transferred through maps fitted on format-matched task-free prompts; Tier 2 = Qwen2.5-7B pairs on M3U96b. Judge in particular: whether the Tier-0 capability (per-step operator map once the trace is visible) supports the paper's claims; whether "causally identified" is earned (interchange + ablation + rescue + sham); the annihilation proposition (regression-fitted maps remove recipient-unpredictable directions) — its hypotheses, proof and the approximate case; the preregistration (what was frozen, the one failed criterion H4 retention ≥ 0.95, Holm family); the control-adjusted estimand against a sham map that itself destroys the control stratum; same- vs different-initialization pairs (seed = init seed); the donor-in-the-loop nature of the primary graft vs the weaker donor-free variants; rule-vs-cache; the strength of the conventional-learning references; Tier-1 selection on the dev split and the confirmatory split; the MLX backend (no per-head hooks; no Transformers equivalence test); verbs vs descriptive/causal evidence. Referee E: full REFORMS checklist.
4+- **Data and compute.** All Tier-0 data are synthetic and regenerated from seeds (split hashes in `experiments/manifests/*.json`); `experiments/registry/runs.jsonl` lists every run; `experiments/summaries/<run>/results.json` + `CONFIRMATORY.md`; per-item outputs and trained models under `artifacts/` are git-ignored (regenerable: `scripts/train_models.py` ≈ 13 min MPS; `scripts/run_transplant.py` ≈ 3 min). Tier-1/2 need Apple Silicon + mlx_lm + the bf16 checkpoints (HF cache on the laptop and M3U96a/b; Qwen2.5-7B on M3U96b). Stage B referees do not run inference: lightweight checks on `experiments/summaries/` only. Stages D–F may re-run Tier-0 experiments on any node with torch (M4M64a) or the laptop, and MLX inference on M3U96a/b replicas (`~/wpai0001_review/`), one GPU job per node.
5+- **No paid API call (0 USD).** None is needed. Crossref/OpenAlex/arXiv lookups are free and allowed.
6+- **Blinding (addition to COMMON.md).** No prior external review exists. `docs/MANDATE.md` and `CLAUDE.md` are blinded by COMMON.md (they are the author's instructions); `docs/{RESEARCH_GAP,PREREGISTRATION,EXPERIMENT_PLAN,COMPUTE_BUDGET,RISK_REGISTER,REPRODUCIBILITY,EXPERIMENT_LOG,DECISION_LOG,TASKS}.md`, `literature/`, `experiments/` are the author's design/audit records: readable.
7+- **Numbers.** `paper/tables/*.tex` are generated by `scripts/build_tables.py` and `paper/figures/*.pdf` by `scripts/build_figures.py` from `experiments/summaries/<run>/results.json` (`paper/figures/FIGURE_PROVENANCE.md` names the run per figure); confirmatory decisions by `scripts/confirmatory_analysis.py` → `CONFIRMATORY.md` — spot-check the paper against these.
8+- Single-author paper (Boucher). Never touch the author's checkout `ia/wp-ai-0001` nor `M3U96b:~/wpai0001`.
added docs/review_enrolment/_ctx.json +7 −0
@@ -0,0 +1,7 @@
1+{
2+ "num": "AI-0001",
3+ "thesis": "in a controlled micro-world (toy decoder transformers trained from scratch on an invented operator language, donor = all 8 operators, recipient = operator Ω excluded), a causally localized subsystem (the post-layer-0 residual row at the read position, identified by interchange interventions, ablation and rescue) can be transplanted at inference time by a donor-in-the-loop activation graft through an orthogonal Procrustes map fitted on task-free rows, lifting the recipient from chance to 0.966 on held-out Ω programs (preregistered confirmatory run, control-adjusted +0.875, 9/9 pairs above the SESOI, control stratum 0.992, one pair below the 0.95 retention criterion), carrying the rule to unseen operands, needing 8 of 128 dimensions and surviving width/depth/stride-2 mismatches; regression-fitted maps (ridge, CCA) annihilate the capability (Proposition 1), donor-free adapters cache, weight grafts cost retention, steering vectors transfer nothing, and conventional learning on the same items is perfect; on open-weight models (Qwen2.5-1.5B/3B, Llama-3.2-3B; Qwen2.5-7B on a node) task vectors are causally sufficient within a model for some tasks and cross-model transfer is reported as measured under a frozen selection rule",
4+ "node": "Tier-0 experiments (PyTorch MPS, ≈ 3 min per transplant run, ≈ 13 min for the 17-model matrix) run on the laptop or on M4M64a (torch 2.12 available; `~/wpai0001_review/` replica via rsync of the worktree, never the author's checkout); Tier-1/2 MLX inference on M3U96a/M3U96b replicas (`~/wpai0001_review/`; checkpoints in the HF cache: Qwen2.5-1.5B/3B, Llama-3.2-3B on both, Qwen2.5-7B on M3U96b; mlx_lm 0.31.3), ONE GPU job per node; detached nohup + log + done-marker, pull back only `experiments/summaries/*` and `experiments/manifests/*` (≤ 20 MB) and commit per package",
5+ "rules": "BINDING (peer_reviews/WP-AI-0001/_constraints.md): 0 USD — no paid LLM/API call (none needed); the frozen PREREGISTRATION text (v1.0 + Tier-1 rule) is never rewritten (dated amendments only in DECISION_LOG); trained models and runs are never deleted (new runs get new ids); never touch the author's checkout ia/wp-ai-0001 nor M3U96b:~/wpai0001",
6+ "ar": "AR-AI0001-1"
7+}
\ No newline at end of file
added docs/review_enrolment/_stage_params.json +13 −0
@@ -0,0 +1,13 @@
1+{
2+ "node": "laptop or M4M64a for Tier 0 (torch); M3U96a/M3U96b for MLX Tier 1/2",
3+ "nodedir": "~/wpai0001_review",
4+ "cache": "HF hub cache on the nodes (bf16 checkpoints); artifacts/models/matrix_v1 regenerable in ≈ 13 min",
5+ "noderules": "Compute: Tier-0 re-runs (`scripts/train_models.py`, `scripts/run_transplant.py`, `scripts/eval_persistent.py`, `scripts/confirmatory_analysis.py`) take minutes on MPS: run them on the laptop (`nice -n 10`) or on M4M64a (`rsync -a --exclude artifacts --exclude .git <worktree>/ M4M64a:~/wpai0001_review/`; torch 2.12.1 present); Tier-1/2 MLX runs (`scripts/run_tier1.py`) on M3U96a or M3U96b replicas with the checkpoints in their HF cache, ONE GPU job per node (`mld scan` first; never M1M32; never the author's `~/wpai0001`); detached via nohup + log + done-marker, poll every 60–120 s, pull back only `experiments/summaries/<run>/` and `experiments/manifests/<run>.json` and commit per package; every new run gets a new run id (never overwrite); the frozen preregistration is never rewritten. 0 USD.",
6+ "lrules": "Batch L (text, bibliography, tables/figures regeneration from committed results, sub-minute checks): laptop allowed (`python3 -I`, `nice -n 10`); anything > 10 min goes to a node replica as for batch N. 0 USD.",
7+ "gen": "paper tables are generated by `scripts/build_tables.py` and figures by `scripts/build_figures.py --run <run id>` from `experiments/summaries/<run>/results.json` (provenance in `paper/figures/FIGURE_PROVENANCE.md`); confirmatory decisions by `scripts/confirmatory_analysis.py` (Tier 0) and `scripts/tier1_analysis.py` (Tier 1/2); claims mapped in `docs/CLAIM_EVIDENCE.md`",
8+ "exhibit_paths": "scripts/*.py (generators and fixes), src/wp_ai_0001/** (fixes with tests), tests/**, experiments/{registry,manifests,summaries}/** (new runs only; never rewriting existing runs), paper/tables/*.tex, paper/figures/*, paper/main.tex ONLY for \\input lines and the exhibit preamble, docs/CLAIM_EVIDENCE.md, README.md exhibit map — PLUS the author's figure mandate of 2026-11-09 (improvements/_FIGURE_MANDATE_2026-11-09.md, binding): one 'cartography' system-map figure (donor → localization → map → graft → evaluation, with the models/nodes/backends used) placed early, every figure legendary-grade (vector, validated palette, uncertainty, units, grayscale-safe, self-contained captions), audited in colour and grayscale",
9+ "writing_paths": "paper/sections/*.tex, paper/appendix/*.tex prose and proofs, the title/subtitle/abstract/version macros of paper/main.tex (\\WPversion → 1.1), paper/references.bib (Stage D additions per improvements/WP-AI-0001/literature_additions.md; verified entries only), README.md prose, docs/{EXPERIMENT_LOG,DECISION_LOG}.md (dated entries only; PREREGISTRATION never rewritten), improvements/WP-AI-0001/response_to_referees.md",
10+ "thesis": "TO FILL after Stage D from improvements/WP-AI-0001/implemented_changes.md",
11+ "verify_numbers": "TO FILL after Stage D",
12+ "repo": "wpai0001_uqo.git"
13+}
\ No newline at end of file
added docs/review_enrolment/enroll_wpai0001.py +34 −0
@@ -0,0 +1,34 @@
1+#!/usr/bin/env python3
2+"""Enrol WP-AI-0001 in the UQO-WP-REVIEW pipeline: registry entry, blinded record, status file, constraints.
3+Computes pages / n_refs / git head at run time. Idempotent (replaces an existing WP-AI-0001 entry)."""
4+import json, os, re, subprocess, datetime
5+ROOT=os.path.expanduser('~/Desktop/UQO/Recherche/Recherche-UQO'); REV=f'{ROOT}/UQO-WP-REVIEW'
6+PID='WP-AI-0001'; PATH='ia/wp-ai-0001'; SRC=f'{ROOT}/{PATH}'
7+def sh(c): return subprocess.check_output(c,shell=True,text=True).strip()
8+head=sh(f'git -C {SRC} rev-parse --short HEAD'); dirty=int(sh(f'git -C {SRC} status --porcelain | wc -l'))
9+pages=int(sh(f'pdfinfo {SRC}/paper/main.pdf | awk \'/^Pages/{{print $2}}\''))
10+nrefs=len(re.findall(r'^@\w+\{', open(f'{SRC}/paper/references.bib').read(), re.M))
11+tex=open(f'{SRC}/paper/main.tex').read()
12+ver=re.search(r'\\newcommand\{\\WPversion\}\{(.*?)\}',tex).group(1); date=re.search(r'\\newcommand\{\\WPdate\}\{(.*?)\}',tex).group(1); jel=re.search(r'\\newcommand\{\\WPjel\}\{(.*?)\}',tex).group(1)
13+title=re.search(r'\\newcommand\{\\WPtitle\}\{(.*?)\}',tex).group(1); sub=re.search(r'\\newcommand\{\\WPsubtitle\}\{(.*?)\}',tex).group(1)
14+now=datetime.datetime.now().astimezone().isoformat(timespec='seconds')
15+readme=open(f'{SRC}/README.md').read(); m=re.search(r'\*\*TL;DR\*\*\s*—\s*(.*?)\n\n',readme,re.S); tldr=re.sub(r'\s*\n>\s*',' ',m.group(1)).strip() if m else ''
16+entry={"id":PID,"number":None,"series":"UQO","alt_ids":["wp-ai-0001","AI-0001"],"title":title,"short_title":"Neural Organ Transplant","subtitle":sub,
17+ "authors":["Simon-Pierre Boucher"],"discipline":"ai","paper_type":"computational","path":PATH,"git_remote":"gitsrv:srv/git/wpai0001_uqo.git","git_head":head,"git_dirty_files":dirty,"git_tags":[],
18+ "baseline_tag":"review-baseline-20261103","review_branch":"review-20261103","worktree":f"UQO-WP-REVIEW/worktrees/{PID}",
19+ "manuscript":{"main_tex":"paper/main.tex","pdf":"paper/main.pdf","pages":pages,"pdf_mtime":datetime.date.today().isoformat(),"engine":"pdflatex (paper/.latexmkrc)","bib":["paper/references.bib"],"n_refs":nrefs,"bib_system":"natbib","version_macro":ver,"date_macro":date,"jel":jel},
20+ "code":{"languages":["python","shell"],"entrypoints":sorted(os.listdir(f'{SRC}/scripts')),"env_files":["pyproject.toml"],"has_makefile":False},
21+ "data":{"summary":"synthetic micro-world splits regenerated from seeds (hashes in manifests); Tier-1 task lists in src; open-weight checkpoints from the HF hub (not committed)","data_dir_mb":0.0,"data_n_files":1,"data_gitignored":True,"raw_preserved":"n/a (synthetic)","sources":"to be completed by Referee D (Stage B)"},
22+ "compute_home":"laptop (M5 Max, MPS + MLX); Tier-2 pairs on M3U96b (~/wpai0001)","status":"complete",
23+ "dissemination":{"ssrn_id":None,"ssrn_status":"not on SSRN","target_journals":[],"submitted":False},
24+ "known_limitations":["Tier-0 capability is a per-step map once the trace is visible; donor-in-the-loop graft; conventional learning perfect on the same items; Tier-1 three models, four tasks"],
25+ "readme_tldr":tldr,"prior_reviews":[],"review_status":{"stage":"0","gate_passed":None,"last_update":now},"checksums_file":f"UQO-WP-REVIEW/manuscripts/{PID}/original/SHA256SUMS","in_scope":True}
26+reg=json.load(open(f'{REV}/registry/working_papers.json'))
27+reg['papers']=[p for p in reg['papers'] if p['id']!=PID]+[entry]; reg['n_in_scope']=sum(1 for p in reg['papers'] if p.get('in_scope')); reg['generated_update']=now
28+json.dump(reg,open(f'{REV}/registry/working_papers.json','w'),indent=1,ensure_ascii=False)
29+# blinded record = entry without review/status/known_limitations/prior_reviews
30+rec={k:v for k,v in entry.items() if k not in ('review_status','status','known_limitations','prior_reviews','in_scope','git_remote','git_dirty_files','git_tags','baseline_tag','review_branch','checksums_file','alt_ids','short_title','readme_tldr')}
31+os.makedirs(f'{REV}/peer_reviews/_briefs/records',exist_ok=True); json.dump(rec,open(f'{REV}/peer_reviews/_briefs/records/{PID}.json','w'),indent=1,ensure_ascii=False)
32+st={"id":PID,"path":PATH,"stage":"0","gate_passed":[],"timestamp":now,"blocking_items":[],"engine_unavailable":[],"api_budget_usd":{"ceiling":0,"spent":0},"notes":[f"{now}: enrolled by the author for the full pipeline (Stages A–G), standing instructions: C–G autonomous, 0 USD, review branch pushed, main untouched, one paper at a time, ≤ 7 concurrent subagents."],"authorization_pending":[],"authorizations":{}}
33+json.dump(st,open(f'{REV}/status/{PID}.json','w'),indent=1)
34+print('enrolled',PID,'head',head,'pages',pages,'refs',nrefs,'version',ver)
added experiments/manifests/tier1_taskvector_confirmatory_625c54bd186f.json +81 −0
@@ -0,0 +1,81 @@
1+{
2+ "config": {
3+ "k": 8,
4+ "localization": "dev-frozen (A2)",
5+ "pairs": [
6+ "qwen3b:qwen1.5b",
7+ "qwen1.5b:qwen3b",
8+ "llama3b:qwen3b",
9+ "qwen3b:llama3b"
10+ ],
11+ "quick": false,
12+ "split": "confirmatory",
13+ "task_seed": 0
14+ },
15+ "created_utc": "2026-11-09T08:44:05+00:00",
16+ "data_hashes": {
17+ "addthree": {
18+ "hash": "cd3d9a8ce6bdbdd9",
19+ "n_queries": 35
20+ },
21+ "antonym": {
22+ "hash": "fd3396e4fae62697",
23+ "n_queries": 32
24+ },
25+ "capital": {
26+ "hash": "e19a1f86c4dae78e",
27+ "n_queries": 22
28+ },
29+ "morph": {
30+ "hash": "09bcd55b41ce2ba7",
31+ "n_queries": 40
32+ }
33+ },
34+ "experiment_id": "tier1_taskvector_confirmatory",
35+ "git": {
36+ "branch": "main",
37+ "commit": "043b146ef4ae8b081c2c6f0246c4c609b04db4c7",
38+ "dirty": true
39+ },
40+ "hardware": {
41+ "chip": "Apple M5 Max",
42+ "cpu_count": 18,
43+ "hostname_hash": "9ad7358c662e",
44+ "machine": "arm64",
45+ "memory_gb": 48.0,
46+ "platform": "macOS-27.0-arm64-arm-64bit-Mach-O",
47+ "torch_cuda": false,
48+ "torch_mps": true
49+ },
50+ "history": [
51+ {
52+ "note": "",
53+ "status": "running",
54+ "utc": "2026-11-09T08:44:08+00:00"
55+ },
56+ {
57+ "note": "",
58+ "status": "completed",
59+ "utc": "2026-11-09T09:06:53+00:00"
60+ }
61+ ],
62+ "interpretation": "",
63+ "libraries": {
64+ "mlx": "?",
65+ "mlx_lm": "0.31.3",
66+ "numpy": "2.4.4",
67+ "pydantic": "2.13.4",
68+ "python": "3.14.4",
69+ "safetensors": "0.7.0",
70+ "scipy": "1.17.1",
71+ "torch": "2.12.0",
72+ "transformers": "5.8.1"
73+ },
74+ "outputs": {
75+ "results": "experiments/summaries/tier1_taskvector_confirmatory_625c54bd186f/results.json"
76+ },
77+ "run_id": "tier1_taskvector_confirmatory_625c54bd186f",
78+ "seeds": {},
79+ "status": "completed",
80+ "warnings": []
81+}
\ No newline at end of file
added experiments/manifests/tier1_taskvector_confirmatory_82a78041c0ce.json +61 −0
@@ -0,0 +1,61 @@
1+{
2+ "config": {
3+ "k": 8,
4+ "pairs": [
5+ "qwen3b:qwen1.5b",
6+ "qwen1.5b:qwen3b",
7+ "llama3b:qwen3b",
8+ "qwen3b:llama3b"
9+ ],
10+ "quick": false,
11+ "split": "confirmatory",
12+ "task_seed": 0
13+ },
14+ "created_utc": "2026-11-09T08:42:55+00:00",
15+ "data_hashes": {},
16+ "experiment_id": "tier1_taskvector_confirmatory",
17+ "git": {
18+ "branch": "main",
19+ "commit": "043b146ef4ae8b081c2c6f0246c4c609b04db4c7",
20+ "dirty": true
21+ },
22+ "hardware": {
23+ "chip": "Apple M5 Max",
24+ "cpu_count": 18,
25+ "hostname_hash": "9ad7358c662e",
26+ "machine": "arm64",
27+ "memory_gb": 48.0,
28+ "platform": "macOS-27.0-arm64-arm-64bit-Mach-O",
29+ "torch_cuda": false,
30+ "torch_mps": true
31+ },
32+ "history": [
33+ {
34+ "note": "",
35+ "status": "running",
36+ "utc": "2026-11-09T08:42:58+00:00"
37+ },
38+ {
39+ "note": "first confirmatory attempt re-localized on confirmatory queries (A2/D13); a relaunch under the same id was killed; superseded by the A2 run id",
40+ "status": "invalidated",
41+ "utc": "2026-11-09T08:55:00+00:00"
42+ }
43+ ],
44+ "interpretation": "",
45+ "libraries": {
46+ "mlx": "?",
47+ "mlx_lm": "0.31.3",
48+ "numpy": "2.4.4",
49+ "pydantic": "2.13.4",
50+ "python": "3.14.4",
51+ "safetensors": "0.7.0",
52+ "scipy": "1.17.1",
53+ "torch": "2.12.0",
54+ "transformers": "5.8.1"
55+ },
56+ "outputs": {},
57+ "run_id": "tier1_taskvector_confirmatory_82a78041c0ce",
58+ "seeds": {},
59+ "status": "invalidated",
60+ "warnings": []
61+}
\ No newline at end of file
modified experiments/manifests/tier1_taskvector_e105e3d91386.json +27 −3
@@ -12,7 +12,24 @@
12 "task_seed": 0 12 "task_seed": 0
13 }, 13 },
14 "created_utc": "2026-11-09T07:46:30+00:00", 14 "created_utc": "2026-11-09T07:46:30+00:00",
15− "data_hashes": {}, 15+ "data_hashes": {
16+ "addthree": {
17+ "hash": "e2d74cd2fbf01431",
18+ "n_queries": 35
19+ },
20+ "antonym": {
21+ "hash": "6f74ac11f9505135",
22+ "n_queries": 32
23+ },
24+ "capital": {
25+ "hash": "78ba5b39d43d42d5",
26+ "n_queries": 22
27+ },
28+ "morph": {
29+ "hash": "df15df1dd69d1028",
30+ "n_queries": 40
31+ }
32+ },
16 "experiment_id": "tier1_taskvector", 33 "experiment_id": "tier1_taskvector",
17 "git": { 34 "git": {
18 "branch": "main", 35 "branch": "main",
@@ -34,6 +51,11 @@
34 "note": "", 51 "note": "",
35 "status": "running", 52 "status": "running",
36 "utc": "2026-11-09T07:46:33+00:00" 53 "utc": "2026-11-09T07:46:33+00:00"
54+ },
55+ {
56+ "note": "",
57+ "status": "completed",
58+ "utc": "2026-11-09T08:14:02+00:00"
37 } 59 }
38 ], 60 ],
39 "interpretation": "", 61 "interpretation": "",
@@ -48,9 +70,11 @@
48 "torch": "2.12.0", 70 "torch": "2.12.0",
49 "transformers": "5.8.1" 71 "transformers": "5.8.1"
50 }, 72 },
51− "outputs": {}, 73+ "outputs": {
74+ "results": "experiments/summaries/tier1_taskvector_e105e3d91386/results.json"
75+ },
52 "run_id": "tier1_taskvector_e105e3d91386", 76 "run_id": "tier1_taskvector_e105e3d91386",
53 "seeds": {}, 77 "seeds": {},
54− "status": "running", 78+ "status": "completed",
55 "warnings": [] 79 "warnings": []
56 } 80 }
\ No newline at end of file
added experiments/manifests/tier1_taskvector_eb5d091bf5ed.json +77 −0
@@ -0,0 +1,77 @@
1+{
2+ "config": {
3+ "k": 8,
4+ "pairs": [
5+ "qwen7b:qwen3b",
6+ "qwen3b:qwen7b",
7+ "qwen7b:llama3b"
8+ ],
9+ "quick": false,
10+ "split": "dev",
11+ "task_seed": 0
12+ },
13+ "created_utc": "2026-11-09T07:54:32+00:00",
14+ "data_hashes": {
15+ "addthree": {
16+ "hash": "e2d74cd2fbf01431",
17+ "n_queries": 35
18+ },
19+ "antonym": {
20+ "hash": "6f74ac11f9505135",
21+ "n_queries": 32
22+ },
23+ "capital": {
24+ "hash": "78ba5b39d43d42d5",
25+ "n_queries": 22
26+ },
27+ "morph": {
28+ "hash": "df15df1dd69d1028",
29+ "n_queries": 40
30+ }
31+ },
32+ "experiment_id": "tier1_taskvector",
33+ "git": {
34+ "branch": null,
35+ "commit": null,
36+ "dirty": null
37+ },
38+ "hardware": {
39+ "chip": "Apple M3 Ultra",
40+ "cpu_count": 28,
41+ "hostname_hash": "1f59c4ad75ba",
42+ "machine": "arm64",
43+ "memory_gb": 96.0,
44+ "platform": "macOS-26.4-arm64-arm-64bit-Mach-O"
45+ },
46+ "history": [
47+ {
48+ "note": "",
49+ "status": "running",
50+ "utc": "2026-11-09T07:54:32+00:00"
51+ },
52+ {
53+ "note": "",
54+ "status": "completed",
55+ "utc": "2026-11-09T08:36:41+00:00"
56+ }
57+ ],
58+ "interpretation": "",
59+ "libraries": {
60+ "mlx": "?",
61+ "mlx_lm": "0.31.3",
62+ "numpy": "2.4.4",
63+ "pydantic": "2.13.4",
64+ "python": "3.14.4",
65+ "safetensors": "0.7.0",
66+ "scipy": "1.17.1",
67+ "torch": null,
68+ "transformers": "5.10.2"
69+ },
70+ "outputs": {
71+ "results": "experiments/summaries/tier1_taskvector_eb5d091bf5ed/results.json"
72+ },
73+ "run_id": "tier1_taskvector_eb5d091bf5ed",
74+ "seeds": {},
75+ "status": "completed",
76+ "warnings": []
77+}
\ No newline at end of file
modified experiments/registry/runs.jsonl +12 −0
@@ -23,3 +23,15 @@
23 {"experiment_id": "graft_tier0_confirmatory", "results": "experiments/summaries/graft_tier0_confirmatory_0c2257f6803e/results.json", "run_id": "graft_tier0_confirmatory_0c2257f6803e", "status": "completed", "utc": "2026-11-09T07:44:03+00:00"} 23 {"experiment_id": "graft_tier0_confirmatory", "results": "experiments/summaries/graft_tier0_confirmatory_0c2257f6803e/results.json", "run_id": "graft_tier0_confirmatory_0c2257f6803e", "status": "completed", "utc": "2026-11-09T07:44:03+00:00"}
24 {"error": "ValueError(\"invalid literal for int() with base 10: 'rocrustes'\")", "experiment_id": "tier1_taskvector", "run_id": "tier1_taskvector_e105e3d91386", "status": "failed", "utc": "2026-11-09T07:45:09+00:00"} 24 {"error": "ValueError(\"invalid literal for int() with base 10: 'rocrustes'\")", "experiment_id": "tier1_taskvector", "run_id": "tier1_taskvector_e105e3d91386", "status": "failed", "utc": "2026-11-09T07:45:09+00:00"}
25 {"config_hash": "e105e3d91386", "experiment_id": "tier1_taskvector", "run_id": "tier1_taskvector_e105e3d91386", "status": "planned", "utc": "2026-11-09T07:46:33+00:00"} 25 {"config_hash": "e105e3d91386", "experiment_id": "tier1_taskvector", "run_id": "tier1_taskvector_e105e3d91386", "status": "planned", "utc": "2026-11-09T07:46:33+00:00"}
26+{"experiment_id": "tier1_taskvector", "run_id": "tier1_taskvector_e105e3d91386", "status": "completed", "utc": "2026-11-09T08:14:02+00:00"}
27+{"config_hash": "82a78041c0ce", "experiment_id": "tier1_taskvector_confirmatory", "run_id": "tier1_taskvector_confirmatory_82a78041c0ce", "status": "planned", "utc": "2026-11-09T08:15:06+00:00"}
28+{"config_hash": "eb5d091bf5ed", "experiment_id": "tier1_taskvector", "run_id": "tier1_taskvector_eb5d091bf5ed", "status": "planned", "utc": "2026-11-09T07:50:22+00:00"}
29+{"error": "ModuleNotFoundError(\"No module named 'torch'\")", "experiment_id": "tier1_taskvector", "run_id": "tier1_taskvector_eb5d091bf5ed", "status": "failed", "utc": "2026-11-09T07:50:22+00:00"}
30+{"config_hash": "eb5d091bf5ed", "experiment_id": "tier1_taskvector", "run_id": "tier1_taskvector_eb5d091bf5ed", "status": "planned", "utc": "2026-11-09T07:51:06+00:00"}
31+{"config_hash": "eb5d091bf5ed", "experiment_id": "tier1_taskvector", "run_id": "tier1_taskvector_eb5d091bf5ed", "status": "planned", "utc": "2026-11-09T07:54:32+00:00"}
32+{"experiment_id": "tier1_taskvector", "run_id": "tier1_taskvector_eb5d091bf5ed", "status": "completed", "utc": "2026-11-09T08:36:41+00:00"}
33+{"experiment_id": "tier1_taskvector_confirmatory", "run_id": "tier1_taskvector_confirmatory_82a78041c0ce", "status": "completed", "utc": "2026-11-09T08:41:48+00:00"}
34+{"utc": "2026-11-09T08:50:00+00:00", "run_id": "tier1_taskvector_confirmatory_82a78041c0ce", "experiment_id": "tier1_taskvector_confirmatory", "status": "invalidated", "note": "layers not frozen from dev (A2/D13)"}
35+{"config_hash": "82a78041c0ce", "experiment_id": "tier1_taskvector_confirmatory", "run_id": "tier1_taskvector_confirmatory_82a78041c0ce", "status": "planned", "utc": "2026-11-09T08:42:58+00:00"}
36+{"config_hash": "625c54bd186f", "experiment_id": "tier1_taskvector_confirmatory", "run_id": "tier1_taskvector_confirmatory_625c54bd186f", "status": "planned", "utc": "2026-11-09T08:44:08+00:00"}
37+{"experiment_id": "tier1_taskvector_confirmatory", "run_id": "tier1_taskvector_confirmatory_625c54bd186f", "status": "completed", "utc": "2026-11-09T09:06:53+00:00"}
added experiments/summaries/tier1_taskvector_confirmatory_625c54bd186f/TIER1_CONFIRMATORY.md +26 −0
@@ -0,0 +1,26 @@
1+# Tier-1/2 confirmatory analysis — dev `tier1_taskvector_e105e3d91386`, confirmatory `tier1_taskvector_confirmatory_625c54bd186f`
2+
3+| pair | task | eligible (own vector dev) | selected condition (dev acc) | conf: zero / few / own / selected / best control | selected − best control [95% CI] |
4+|---|---|---|---|---|---|
5+| qwen3b->qwen1.5b | antonym | yes (0.88) | replace_donor_fewshot|procrustes|L16 (0.72) | 0.19 / 0.94 / 0.91 / **0.59** / 0.44 (random_orth|L19|a1.0) | +0.16 [+0.00, +0.34]; pre-A1 control 0.69; shuffled-query diag. 0.69 |
6+| qwen3b->qwen1.5b | capital | yes (0.86) | ridge|L22|a1.0 (0.18) | 0.00 / 1.00 / 0.82 / **0.09** / 0.00 (random_orth|L19|a1.0) | +0.09 [+0.00, +0.23]; pre-A1 control 0.00; shuffled-query diag. 0.00 |
7+| qwen3b->qwen1.5b | morph | no (0.00) | procrustes|L2|a1.0 (0.00) | 0.00 / 0.30 / 0.00 / **0.00** / 0.00 (random_orth|L2|a1.0) | +0.00 [+0.00, +0.00]; pre-A1 control 0.00; shuffled-query diag. 0.00 |
8+| qwen3b->qwen1.5b | addthree | no (0.00) | procrustes|L2|a1.0 (0.00) | 0.00 / 0.86 / 0.00 / **0.00** / 0.00 (random_orth|L2|a1.0) | +0.00 [+0.00, +0.00]; pre-A1 control 0.00; shuffled-query diag. 0.00 |
9+| **qwen3b->qwen1.5b** | | | | | **H6 not supported (0 task(s))** |
10+| qwen1.5b->qwen3b | antonym | yes (0.88) | procrustes_rescaled|L21|a1.0 (0.75) | 0.09 / 0.94 / 0.91 / **0.69** / 0.41 (ctl_unrelated_task_vector(capital)|L25) | +0.28 [+0.09, +0.47]; pre-A1 control 0.81; shuffled-query diag. 0.81 |
11+| qwen1.5b->qwen3b | capital | yes (0.82) | procrustes|L28|a1.0 (0.73) | 0.14 / 1.00 / 0.82 / **0.55** / 0.18 (ctl_unrelated_task_vector(antonym)|L28) | +0.36 [+0.14, +0.59] ✔; pre-A1 control 0.27; shuffled-query diag. 0.27 |
12+| qwen1.5b->qwen3b | morph | no (0.00) | procrustes|L2|a1.0 (0.00) | 0.00 / 0.40 / 0.00 / **0.00** / 0.00 (random_orth|L2|a1.0) | +0.00 [+0.00, +0.00]; pre-A1 control 0.00; shuffled-query diag. 0.00 |
13+| qwen1.5b->qwen3b | addthree | no (0.00) | procrustes|L2|a1.0 (0.00) | 0.00 / 1.00 / 0.00 / **0.00** / 0.00 (random_orth|L2|a1.0) | +0.00 [+0.00, +0.00]; pre-A1 control 0.00; shuffled-query diag. 0.00 |
14+| **qwen1.5b->qwen3b** | | | | | **H6 partial (1 task(s))** |
15+| llama3b->qwen3b | antonym | yes (0.88) | cca|L25|a1.0 (0.81) | 0.09 / 0.94 / 0.91 / **0.81** / 0.31 (ctl_unrelated_task_vector(capital)|L25) | +0.50 [+0.31, +0.69] ✔; pre-A1 control 0.81; shuffled-query diag. 0.81 |
16+| llama3b->qwen3b | capital | yes (0.82) | cca|L18|a1.0 (0.50) | 0.14 / 1.00 / 0.82 / **0.64** / 0.45 (ctl_replace_donor_zeroshot|sham_map|L18) | +0.18 [+0.05, +0.36]; pre-A1 control 0.45; shuffled-query diag. 0.41 |
17+| llama3b->qwen3b | morph | no (0.00) | procrustes|L2|a1.0 (0.00) | 0.00 / 0.40 / 0.00 / **0.00** / 0.00 (random_orth|L2|a1.0) | +0.00 [+0.00, +0.00]; pre-A1 control 0.00; shuffled-query diag. 0.00 |
18+| llama3b->qwen3b | addthree | no (0.00) | procrustes|L2|a1.0 (0.00) | 0.00 / 1.00 / 0.00 / **0.00** / 0.00 (random_orth|L2|a1.0) | +0.00 [+0.00, +0.00]; pre-A1 control 0.00; shuffled-query diag. 0.00 |
19+| **llama3b->qwen3b** | | | | | **H6 partial (1 task(s))** |
20+| qwen3b->llama3b | antonym | yes (0.84) | procrustes|L14|a1.0 (0.66) | 0.06 / 0.91 / 0.84 / **0.59** / 0.34 (ctl_unrelated_task_vector(capital)|L19) | +0.25 [+0.09, +0.41]; pre-A1 control 0.38; shuffled-query diag. 0.38 |
21+| qwen3b->llama3b | capital | yes (1.00) | procrustes|L14|a1.0 (0.05) | 0.09 / 1.00 / 1.00 / **0.09** / 0.14 (ctl_unrelated_task_vector(antonym)|L14) | -0.05 [-0.18, +0.09]; pre-A1 control 0.14; shuffled-query diag. 0.09 |
22+| qwen3b->llama3b | morph | no (0.00) | procrustes|L2|a1.0 (0.00) | 0.00 / 0.28 / 0.00 / **0.00** / 0.00 (random_orth|L2|a1.0) | +0.00 [+0.00, +0.00]; pre-A1 control 0.00; shuffled-query diag. 0.00 |
23+| qwen3b->llama3b | addthree | yes (0.80) | procrustes|L2|a1.0 (0.00) | 0.00 / 1.00 / 0.69 / **0.00** / 0.00 (random_orth|L2|a1.0) | +0.00 [+0.00, +0.00]; pre-A1 control 0.00; shuffled-query diag. 0.00 |
24+| **qwen3b->llama3b** | | | | | **H6 not supported (0 task(s))** |
25+
26+Note: CIs are exact paired item bootstraps (2 000 draws) on per-item correctness stored in the confirmatory results.
added experiments/summaries/tier1_taskvector_confirmatory_625c54bd186f/results.json +19597 −0
Large diff (19,597 lines) is not loaded by default to keep the page fast.Load diff

Diff truncated — file too large.

Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere