SPB Git
spboucher.ai

spb/wpai0001_uqo

Public

Tier-1/2 runs (dev + confirmatory, amendments A1-A2), frozen Tier-1 analysis, paper v1.0 draft (Tier-2 confirmatory paragraph pending)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Nov 9, 2026) parent 043b146 Browse files

14 changed files +19,967 −3

14 changed files
Load all 14 files Unified Split
modified docs/DECISION_LOG.md +4 −0
@@ -44,3 +44,7 @@ All entries are dated; "before freeze" refers to the confirmatory freeze (PREREG
44 44 PREREGISTRATION before the development results were read (eligibility = own-vector sufficiency ≥ 0.5;
45 45 selected condition = best dev condition; contrast = selected minus best control on confirmatory queries).
46 46 PREREGISTRATION.md sha256 at Tier-1 freeze: 85c5db69657144e6d6afecfb4fc346c2345d09455fb6134bee615567e5c705ad
47 +- **D12 (2026-11-09, amendment A1 before the Tier-1 confirmatory run).** Shuffled-query donor state reclassified
48 + from null control to mechanism diagnostic (it carries the task identity); both analyses reported.
49 +- **D13 (2026-11-09, amendment A2).** Tier-1/2 confirmatory runs load the development localization (layers,
50 + vectors) instead of re-localizing on confirmatory queries; first confirmatory run invalidated and re-run.
modified docs/EXPERIMENT_LOG.md +4 −0
@@ -83,3 +83,7 @@ mechanism unchanged (ablation 0.086 → rescue 1.000, sham 0.115; reverse 0.965)
83 83
84 84 | 2026-11-09 | `graft_tier0_confirmatory_0c2257f6803e` (MPS, PREREGISTRATION v1.0, seed base 90 000) | **completed** | `experiments/summaries/graft_tier0_confirmatory_0c2257f6803e/{results.json,CONFIRMATORY.md,confirmatory.json}` | primary: 9/9 pairs lower bound > SESOI, mean Δ_T^adj 0.875 (sd 0.042, min 0.776) → H2 SUPPORTED; H4 NOT SUPPORTED (one pair control 0.946 < 0.95; others ≥ 0.985); Holm: subspace k8 vs random +0.850, adapter vs base +0.472, weight L0 block vs shuffled +0.556 (retention 0.720), all rejected at Holm; H5 seen 0.941 / unseen 0.768 (ratio 0.816 ≥ 0.8) SUPPORTED; adapter cache 0.702 / 0.000 |
85 85 | 2026-11-09 | `eval_persistent` on the confirmatory run (fresh process, seed base 95 000) | completed | `.../persistent/fresh_process_eval.json` | saved adapter: target 0.344 [0.317, 0.372] vs base 0.115; control 0.979 |
86 +| 2026-11-09 | `tier1_taskvector_e105e3d91386` (MLX laptop, dev split, 4 pairs, k = 8, 10 vector sets) | completed (1 649 s) | `experiments/summaries/tier1_taskvector_e105e3d91386/results.json` | within-model sufficiency: antonym 0.84–0.88 (L14–L25), capital 0.82–1.00 (L14–L25), addthree only in Llama (0.80, L14), morph none; cross-model best (dev): antonym 0.66–0.81 in all 4 pairs vs best null control 0.38–0.56; capital 0.73 (1.5B→3B, Procrustes L28), 0.50 (Llama→Qwen3B, CCA L18), 0.18 (3B→1.5B), 0.05 (3B→Llama); shuffled-query donor state transfers antonym as well as the matched query (→ amendment A1, D12) |
87 +| 2026-11-09 | `tier1_taskvector_eb5d091bf5ed` (MLX on M3U96b, Tier 2: Qwen2.5-7B ↔ Qwen2.5-3B, Qwen2.5-7B → Llama-3.2-3B, dev split) | completed (2 528 s) | `experiments/summaries/tier1_taskvector_eb5d091bf5ed/results.json` (pulled from the node) | Qwen-7B within-model: antonym 0.91 (L16), capital 1.00 (L11), morph 0.03, add-three 0.00; dev transfer: antonym 0.84 both Qwen directions (controls 0.53/0.59), capital 0.91 (3B→7B; unrelated-task control 0.77) / 0.14 (7B→3B); 7B→Llama antonym 0.38 vs control 0.34 (no transfer) |
88 +| 2026-11-09 | `tier1_taskvector_confirmatory_82a78041c0ce` | **invalidated** | manifest | re-localized on confirmatory queries (A2/D13); no number used |
89 +| 2026-11-09 | `tier1_taskvector_confirmatory_625c54bd186f` (laptop, confirmatory split, dev-frozen localization) | completed | `experiments/summaries/tier1_taskvector_confirmatory_625c54bd186f/{results.json,TIER1_CONFIRMATORY.md}` | H6 per pair (frozen rule, amended controls): Qwen3B→1.5B not supported (antonym +0.16 [0.00, 0.34], capital +0.09); 1.5B→3B partial (capital +0.36 [0.14, 0.59] ✔, antonym +0.28 [0.09, 0.47]); Llama→Qwen3B partial (antonym +0.50 [0.31, 0.69] ✔, capital +0.18 [0.05, 0.36]); Qwen3B→Llama not supported (antonym +0.25 [0.09, 0.41], capital −0.05); add-three/morph not eligible or 0 |
modified docs/PREREGISTRATION.md +16 −0
@@ -130,3 +130,19 @@ script's conditions.
130 130 "partial" if for one; "not supported" otherwise. Also reported: gain vs recipient zero-shot, recipient few-shot
131 131 (prompting ceiling), recipient own vector (within-model ceiling), held-out NLL change and cross-task accuracy.
132 132 - Tier 2 (Qwen2.5-7B pairs, run on M3U96b) follows the same rule.
133 +
134 +### Amendment A1 to the Tier-1 rule (2026-11-09, recorded BEFORE the Tier-1 confirmatory run; D12)
135 +The first development pair showed that the "mapped donor few-shot state of a *shuffled* query" condition is not a
136 +null control: the recipient keeps its own query in context, and the donor state carries the task identity, so it
137 +transfers the task exactly like the matched-query state (antonym, qwen3b→qwen1.5b: 0.72 vs 0.72 on dev). It is
138 +reclassified as a **mechanism diagnostic** (task identity vs query content). The null-control set for the H6
139 +contrast is therefore: norm-matched random vector, unrelated-task vector, sham-map vector (random orthogonal map),
140 +and mapped donor **zero-shot** state (same map, no task information). The analysis reports both the pre-amendment
141 +(shuffled query included) and the amended contrast.
142 +
143 +### Amendment A2 (2026-11-09, procedural; D13)
144 +The first confirmatory Tier-1 run (`tier1_taskvector_confirmatory_82a78041c0ce`) re-localized the recipient's
145 +best layer on the confirmatory queries, so some development-selected conditions did not exist in it. As the rule
146 +intended, localization (layers, vectors) is frozen from the development split; the run is marked *invalidated*
147 +in the registry and re-executed with `--split confirmatory` loading the development localization caches. No
148 +numbers from the invalidated run are used.
added docs/review_enrolment/_constraints_WP-AI-0001.md +8 −0
@@ -0,0 +1,8 @@
1 +## Paper-specific constraints (binding for every stage of this review)
2 +- **Enrolment.** WP-AI-0001 was built autonomously on 2026-11-09 (sprints 0–7 in one session) and enrolled the same day for the full pipeline (Stages A–G) under the standing instructions of the earlier waves: Stages C–G run autonomously; every authorization request is treated as authorised EXCEPT paid API spend (0 USD); the review branch is pushed after every commit; `main` is never touched; one paper at a time; ≤ 7 concurrent subagents.
3 +- **Genre.** Mechanistic-interpretability / machine-learning methods paper with a preregistered confirmatory test: can a causally localized subsystem ("organ", a metaphor) be transplanted from a donor language model into a recipient that lacks the capability, without training? Tier 0 = toy decoder transformers (0.80 M params, 4 layers) trained from scratch on an invented operator micro-world (OPSEQ, scratchpad trace, modulus 11), donor = all 8 operators, recipient = operator Ω excluded; localization by interchange interventions; six transplant families (activation, subspace, steering, adapter, weight, persistent) against a control battery (sham map, norm-matched random, shuffled item/layer, unrelated-skill donor) and conventional-learning references (supervised adapter, distillation); PREREGISTRATION v1.0 frozen before the confirmatory run (`graft_tier0_confirmatory_0c2257f6803e`, seed base 90 000); Tier 1 = Qwen2.5-1.5B/3B-Instruct and Llama-3.2-3B-Instruct in bf16 under MLX, task vectors (few-shot − zero-shot last-token state) localized within model and transferred through maps fitted on format-matched task-free prompts; Tier 2 = Qwen2.5-7B pairs on M3U96b. Judge in particular: whether the Tier-0 capability (per-step operator map once the trace is visible) supports the paper's claims; whether "causally identified" is earned (interchange + ablation + rescue + sham); the annihilation proposition (regression-fitted maps remove recipient-unpredictable directions) — its hypotheses, proof and the approximate case; the preregistration (what was frozen, the one failed criterion H4 retention ≥ 0.95, Holm family); the control-adjusted estimand against a sham map that itself destroys the control stratum; same- vs different-initialization pairs (seed = init seed); the donor-in-the-loop nature of the primary graft vs the weaker donor-free variants; rule-vs-cache; the strength of the conventional-learning references; Tier-1 selection on the dev split and the confirmatory split; the MLX backend (no per-head hooks; no Transformers equivalence test); verbs vs descriptive/causal evidence. Referee E: full REFORMS checklist.
4 +- **Data and compute.** All Tier-0 data are synthetic and regenerated from seeds (split hashes in `experiments/manifests/*.json`); `experiments/registry/runs.jsonl` lists every run; `experiments/summaries/<run>/results.json` + `CONFIRMATORY.md`; per-item outputs and trained models under `artifacts/` are git-ignored (regenerable: `scripts/train_models.py` ≈ 13 min MPS; `scripts/run_transplant.py` ≈ 3 min). Tier-1/2 need Apple Silicon + mlx_lm + the bf16 checkpoints (HF cache on the laptop and M3U96a/b; Qwen2.5-7B on M3U96b). Stage B referees do not run inference: lightweight checks on `experiments/summaries/` only. Stages D–F may re-run Tier-0 experiments on any node with torch (M4M64a) or the laptop, and MLX inference on M3U96a/b replicas (`~/wpai0001_review/`), one GPU job per node.
5 +- **No paid API call (0 USD).** None is needed. Crossref/OpenAlex/arXiv lookups are free and allowed.
6 +- **Blinding (addition to COMMON.md).** No prior external review exists. `docs/MANDATE.md` and `CLAUDE.md` are blinded by COMMON.md (they are the author's instructions); `docs/{RESEARCH_GAP,PREREGISTRATION,EXPERIMENT_PLAN,COMPUTE_BUDGET,RISK_REGISTER,REPRODUCIBILITY,EXPERIMENT_LOG,DECISION_LOG,TASKS}.md`, `literature/`, `experiments/` are the author's design/audit records: readable.
7 +- **Numbers.** `paper/tables/*.tex` are generated by `scripts/build_tables.py` and `paper/figures/*.pdf` by `scripts/build_figures.py` from `experiments/summaries/<run>/results.json` (`paper/figures/FIGURE_PROVENANCE.md` names the run per figure); confirmatory decisions by `scripts/confirmatory_analysis.py` → `CONFIRMATORY.md` — spot-check the paper against these.
8 +- Single-author paper (Boucher). Never touch the author's checkout `ia/wp-ai-0001` nor `M3U96b:~/wpai0001`.
added docs/review_enrolment/_ctx.json +7 −0
@@ -0,0 +1,7 @@
1 +{
2 + "num": "AI-0001",
3 + "thesis": "in a controlled micro-world (toy decoder transformers trained from scratch on an invented operator language, donor = all 8 operators, recipient = operator Ω excluded), a causally localized subsystem (the post-layer-0 residual row at the read position, identified by interchange interventions, ablation and rescue) can be transplanted at inference time by a donor-in-the-loop activation graft through an orthogonal Procrustes map fitted on task-free rows, lifting the recipient from chance to 0.966 on held-out Ω programs (preregistered confirmatory run, control-adjusted +0.875, 9/9 pairs above the SESOI, control stratum 0.992, one pair below the 0.95 retention criterion), carrying the rule to unseen operands, needing 8 of 128 dimensions and surviving width/depth/stride-2 mismatches; regression-fitted maps (ridge, CCA) annihilate the capability (Proposition 1), donor-free adapters cache, weight grafts cost retention, steering vectors transfer nothing, and conventional learning on the same items is perfect; on open-weight models (Qwen2.5-1.5B/3B, Llama-3.2-3B; Qwen2.5-7B on a node) task vectors are causally sufficient within a model for some tasks and cross-model transfer is reported as measured under a frozen selection rule",
4 + "node": "Tier-0 experiments (PyTorch MPS, ≈ 3 min per transplant run, ≈ 13 min for the 17-model matrix) run on the laptop or on M4M64a (torch 2.12 available; `~/wpai0001_review/` replica via rsync of the worktree, never the author's checkout); Tier-1/2 MLX inference on M3U96a/M3U96b replicas (`~/wpai0001_review/`; checkpoints in the HF cache: Qwen2.5-1.5B/3B, Llama-3.2-3B on both, Qwen2.5-7B on M3U96b; mlx_lm 0.31.3), ONE GPU job per node; detached nohup + log + done-marker, pull back only `experiments/summaries/*` and `experiments/manifests/*` (≤ 20 MB) and commit per package",
5 + "rules": "BINDING (peer_reviews/WP-AI-0001/_constraints.md): 0 USD — no paid LLM/API call (none needed); the frozen PREREGISTRATION text (v1.0 + Tier-1 rule) is never rewritten (dated amendments only in DECISION_LOG); trained models and runs are never deleted (new runs get new ids); never touch the author's checkout ia/wp-ai-0001 nor M3U96b:~/wpai0001",
6 + "ar": "AR-AI0001-1"
7 +}
\ No newline at end of file
added docs/review_enrolment/_stage_params.json +13 −0
@@ -0,0 +1,13 @@
1 +{
2 + "node": "laptop or M4M64a for Tier 0 (torch); M3U96a/M3U96b for MLX Tier 1/2",
3 + "nodedir": "~/wpai0001_review",
4 + "cache": "HF hub cache on the nodes (bf16 checkpoints); artifacts/models/matrix_v1 regenerable in ≈ 13 min",
5 + "noderules": "Compute: Tier-0 re-runs (`scripts/train_models.py`, `scripts/run_transplant.py`, `scripts/eval_persistent.py`, `scripts/confirmatory_analysis.py`) take minutes on MPS: run them on the laptop (`nice -n 10`) or on M4M64a (`rsync -a --exclude artifacts --exclude .git <worktree>/ M4M64a:~/wpai0001_review/`; torch 2.12.1 present); Tier-1/2 MLX runs (`scripts/run_tier1.py`) on M3U96a or M3U96b replicas with the checkpoints in their HF cache, ONE GPU job per node (`mld scan` first; never M1M32; never the author's `~/wpai0001`); detached via nohup + log + done-marker, poll every 60–120 s, pull back only `experiments/summaries/<run>/` and `experiments/manifests/<run>.json` and commit per package; every new run gets a new run id (never overwrite); the frozen preregistration is never rewritten. 0 USD.",
6 + "lrules": "Batch L (text, bibliography, tables/figures regeneration from committed results, sub-minute checks): laptop allowed (`python3 -I`, `nice -n 10`); anything > 10 min goes to a node replica as for batch N. 0 USD.",
7 + "gen": "paper tables are generated by `scripts/build_tables.py` and figures by `scripts/build_figures.py --run <run id>` from `experiments/summaries/<run>/results.json` (provenance in `paper/figures/FIGURE_PROVENANCE.md`); confirmatory decisions by `scripts/confirmatory_analysis.py` (Tier 0) and `scripts/tier1_analysis.py` (Tier 1/2); claims mapped in `docs/CLAIM_EVIDENCE.md`",
8 + "exhibit_paths": "scripts/*.py (generators and fixes), src/wp_ai_0001/** (fixes with tests), tests/**, experiments/{registry,manifests,summaries}/** (new runs only; never rewriting existing runs), paper/tables/*.tex, paper/figures/*, paper/main.tex ONLY for \\input lines and the exhibit preamble, docs/CLAIM_EVIDENCE.md, README.md exhibit map — PLUS the author's figure mandate of 2026-11-09 (improvements/_FIGURE_MANDATE_2026-11-09.md, binding): one 'cartography' system-map figure (donor → localization → map → graft → evaluation, with the models/nodes/backends used) placed early, every figure legendary-grade (vector, validated palette, uncertainty, units, grayscale-safe, self-contained captions), audited in colour and grayscale",
9 + "writing_paths": "paper/sections/*.tex, paper/appendix/*.tex prose and proofs, the title/subtitle/abstract/version macros of paper/main.tex (\\WPversion → 1.1), paper/references.bib (Stage D additions per improvements/WP-AI-0001/literature_additions.md; verified entries only), README.md prose, docs/{EXPERIMENT_LOG,DECISION_LOG}.md (dated entries only; PREREGISTRATION never rewritten), improvements/WP-AI-0001/response_to_referees.md",
10 + "thesis": "TO FILL after Stage D from improvements/WP-AI-0001/implemented_changes.md",
11 + "verify_numbers": "TO FILL after Stage D",
12 + "repo": "wpai0001_uqo.git"
13 +}
\ No newline at end of file
added docs/review_enrolment/enroll_wpai0001.py +34 −0
@@ -0,0 +1,34 @@
1 +#!/usr/bin/env python3
2 +"""Enrol WP-AI-0001 in the UQO-WP-REVIEW pipeline: registry entry, blinded record, status file, constraints.
3 +Computes pages / n_refs / git head at run time. Idempotent (replaces an existing WP-AI-0001 entry)."""
4 +import json, os, re, subprocess, datetime
5 +ROOT=os.path.expanduser('~/Desktop/UQO/Recherche/Recherche-UQO'); REV=f'{ROOT}/UQO-WP-REVIEW'
6 +PID='WP-AI-0001'; PATH='ia/wp-ai-0001'; SRC=f'{ROOT}/{PATH}'
7 +def sh(c): return subprocess.check_output(c,shell=True,text=True).strip()
8 +head=sh(f'git -C {SRC} rev-parse --short HEAD'); dirty=int(sh(f'git -C {SRC} status --porcelain | wc -l'))
9 +pages=int(sh(f'pdfinfo {SRC}/paper/main.pdf | awk \'/^Pages/{{print $2}}\''))
10 +nrefs=len(re.findall(r'^@\w+\{', open(f'{SRC}/paper/references.bib').read(), re.M))
11 +tex=open(f'{SRC}/paper/main.tex').read()
12 +ver=re.search(r'\\newcommand\{\\WPversion\}\{(.*?)\}',tex).group(1); date=re.search(r'\\newcommand\{\\WPdate\}\{(.*?)\}',tex).group(1); jel=re.search(r'\\newcommand\{\\WPjel\}\{(.*?)\}',tex).group(1)
13 +title=re.search(r'\\newcommand\{\\WPtitle\}\{(.*?)\}',tex).group(1); sub=re.search(r'\\newcommand\{\\WPsubtitle\}\{(.*?)\}',tex).group(1)
14 +now=datetime.datetime.now().astimezone().isoformat(timespec='seconds')
15 +readme=open(f'{SRC}/README.md').read(); m=re.search(r'\*\*TL;DR\*\*\s*—\s*(.*?)\n\n',readme,re.S); tldr=re.sub(r'\s*\n>\s*',' ',m.group(1)).strip() if m else ''
16 +entry={"id":PID,"number":None,"series":"UQO","alt_ids":["wp-ai-0001","AI-0001"],"title":title,"short_title":"Neural Organ Transplant","subtitle":sub,
17 + "authors":["Simon-Pierre Boucher"],"discipline":"ai","paper_type":"computational","path":PATH,"git_remote":"gitsrv:srv/git/wpai0001_uqo.git","git_head":head,"git_dirty_files":dirty,"git_tags":[],
18 + "baseline_tag":"review-baseline-20261103","review_branch":"review-20261103","worktree":f"UQO-WP-REVIEW/worktrees/{PID}",
19 + "manuscript":{"main_tex":"paper/main.tex","pdf":"paper/main.pdf","pages":pages,"pdf_mtime":datetime.date.today().isoformat(),"engine":"pdflatex (paper/.latexmkrc)","bib":["paper/references.bib"],"n_refs":nrefs,"bib_system":"natbib","version_macro":ver,"date_macro":date,"jel":jel},
20 + "code":{"languages":["python","shell"],"entrypoints":sorted(os.listdir(f'{SRC}/scripts')),"env_files":["pyproject.toml"],"has_makefile":False},
21 + "data":{"summary":"synthetic micro-world splits regenerated from seeds (hashes in manifests); Tier-1 task lists in src; open-weight checkpoints from the HF hub (not committed)","data_dir_mb":0.0,"data_n_files":1,"data_gitignored":True,"raw_preserved":"n/a (synthetic)","sources":"to be completed by Referee D (Stage B)"},
22 + "compute_home":"laptop (M5 Max, MPS + MLX); Tier-2 pairs on M3U96b (~/wpai0001)","status":"complete",
23 + "dissemination":{"ssrn_id":None,"ssrn_status":"not on SSRN","target_journals":[],"submitted":False},
24 + "known_limitations":["Tier-0 capability is a per-step map once the trace is visible; donor-in-the-loop graft; conventional learning perfect on the same items; Tier-1 three models, four tasks"],
25 + "readme_tldr":tldr,"prior_reviews":[],"review_status":{"stage":"0","gate_passed":None,"last_update":now},"checksums_file":f"UQO-WP-REVIEW/manuscripts/{PID}/original/SHA256SUMS","in_scope":True}
26 +reg=json.load(open(f'{REV}/registry/working_papers.json'))
27 +reg['papers']=[p for p in reg['papers'] if p['id']!=PID]+[entry]; reg['n_in_scope']=sum(1 for p in reg['papers'] if p.get('in_scope')); reg['generated_update']=now
28 +json.dump(reg,open(f'{REV}/registry/working_papers.json','w'),indent=1,ensure_ascii=False)
29 +# blinded record = entry without review/status/known_limitations/prior_reviews
30 +rec={k:v for k,v in entry.items() if k not in ('review_status','status','known_limitations','prior_reviews','in_scope','git_remote','git_dirty_files','git_tags','baseline_tag','review_branch','checksums_file','alt_ids','short_title','readme_tldr')}
31 +os.makedirs(f'{REV}/peer_reviews/_briefs/records',exist_ok=True); json.dump(rec,open(f'{REV}/peer_reviews/_briefs/records/{PID}.json','w'),indent=1,ensure_ascii=False)
32 +st={"id":PID,"path":PATH,"stage":"0","gate_passed":[],"timestamp":now,"blocking_items":[],"engine_unavailable":[],"api_budget_usd":{"ceiling":0,"spent":0},"notes":[f"{now}: enrolled by the author for the full pipeline (Stages A–G), standing instructions: C–G autonomous, 0 USD, review branch pushed, main untouched, one paper at a time, ≤ 7 concurrent subagents."],"authorization_pending":[],"authorizations":{}}
33 +json.dump(st,open(f'{REV}/status/{PID}.json','w'),indent=1)
34 +print('enrolled',PID,'head',head,'pages',pages,'refs',nrefs,'version',ver)
added experiments/manifests/tier1_taskvector_confirmatory_625c54bd186f.json +81 −0
@@ -0,0 +1,81 @@
1 +{
2 + "config": {
3 + "k": 8,
4 + "localization": "dev-frozen (A2)",
5 + "pairs": [
6 + "qwen3b:qwen1.5b",
7 + "qwen1.5b:qwen3b",
8 + "llama3b:qwen3b",
9 + "qwen3b:llama3b"
10 + ],
11 + "quick": false,
12 + "split": "confirmatory",
13 + "task_seed": 0
14 + },
15 + "created_utc": "2026-11-09T08:44:05+00:00",
16 + "data_hashes": {
17 + "addthree": {
18 + "hash": "cd3d9a8ce6bdbdd9",
19 + "n_queries": 35
20 + },
21 + "antonym": {
22 + "hash": "fd3396e4fae62697",
23 + "n_queries": 32
24 + },
25 + "capital": {
26 + "hash": "e19a1f86c4dae78e",
27 + "n_queries": 22
28 + },
29 + "morph": {
30 + "hash": "09bcd55b41ce2ba7",
31 + "n_queries": 40
32 + }
33 + },
34 + "experiment_id": "tier1_taskvector_confirmatory",
35 + "git": {
36 + "branch": "main",
37 + "commit": "043b146ef4ae8b081c2c6f0246c4c609b04db4c7",
38 + "dirty": true
39 + },
40 + "hardware": {
41 + "chip": "Apple M5 Max",
42 + "cpu_count": 18,
43 + "hostname_hash": "9ad7358c662e",
44 + "machine": "arm64",
45 + "memory_gb": 48.0,
46 + "platform": "macOS-27.0-arm64-arm-64bit-Mach-O",
47 + "torch_cuda": false,
48 + "torch_mps": true
49 + },
50 + "history": [
51 + {
52 + "note": "",
53 + "status": "running",
54 + "utc": "2026-11-09T08:44:08+00:00"
55 + },
56 + {
57 + "note": "",
58 + "status": "completed",
59 + "utc": "2026-11-09T09:06:53+00:00"
60 + }
61 + ],
62 + "interpretation": "",
63 + "libraries": {
64 + "mlx": "?",
65 + "mlx_lm": "0.31.3",
66 + "numpy": "2.4.4",
67 + "pydantic": "2.13.4",
68 + "python": "3.14.4",
69 + "safetensors": "0.7.0",
70 + "scipy": "1.17.1",
71 + "torch": "2.12.0",
72 + "transformers": "5.8.1"
73 + },
74 + "outputs": {
75 + "results": "experiments/summaries/tier1_taskvector_confirmatory_625c54bd186f/results.json"
76 + },
77 + "run_id": "tier1_taskvector_confirmatory_625c54bd186f",
78 + "seeds": {},
79 + "status": "completed",
80 + "warnings": []
81 +}
\ No newline at end of file
added experiments/manifests/tier1_taskvector_confirmatory_82a78041c0ce.json +61 −0
@@ -0,0 +1,61 @@
1 +{
2 + "config": {
3 + "k": 8,
4 + "pairs": [
5 + "qwen3b:qwen1.5b",
6 + "qwen1.5b:qwen3b",
7 + "llama3b:qwen3b",
8 + "qwen3b:llama3b"
9 + ],
10 + "quick": false,
11 + "split": "confirmatory",
12 + "task_seed": 0
13 + },
14 + "created_utc": "2026-11-09T08:42:55+00:00",
15 + "data_hashes": {},
16 + "experiment_id": "tier1_taskvector_confirmatory",
17 + "git": {
18 + "branch": "main",
19 + "commit": "043b146ef4ae8b081c2c6f0246c4c609b04db4c7",
20 + "dirty": true
21 + },
22 + "hardware": {
23 + "chip": "Apple M5 Max",
24 + "cpu_count": 18,
25 + "hostname_hash": "9ad7358c662e",
26 + "machine": "arm64",
27 + "memory_gb": 48.0,
28 + "platform": "macOS-27.0-arm64-arm-64bit-Mach-O",
29 + "torch_cuda": false,
30 + "torch_mps": true
31 + },
32 + "history": [
33 + {
34 + "note": "",
35 + "status": "running",
36 + "utc": "2026-11-09T08:42:58+00:00"
37 + },
38 + {
39 + "note": "first confirmatory attempt re-localized on confirmatory queries (A2/D13); a relaunch under the same id was killed; superseded by the A2 run id",
40 + "status": "invalidated",
41 + "utc": "2026-11-09T08:55:00+00:00"
42 + }
43 + ],
44 + "interpretation": "",
45 + "libraries": {
46 + "mlx": "?",
47 + "mlx_lm": "0.31.3",
48 + "numpy": "2.4.4",
49 + "pydantic": "2.13.4",
50 + "python": "3.14.4",
51 + "safetensors": "0.7.0",
52 + "scipy": "1.17.1",
53 + "torch": "2.12.0",
54 + "transformers": "5.8.1"
55 + },
56 + "outputs": {},
57 + "run_id": "tier1_taskvector_confirmatory_82a78041c0ce",
58 + "seeds": {},
59 + "status": "invalidated",
60 + "warnings": []
61 +}
\ No newline at end of file
modified experiments/manifests/tier1_taskvector_e105e3d91386.json +27 −3
@@ -12,7 +12,24 @@
12 12 "task_seed": 0
13 13 },
14 14 "created_utc": "2026-11-09T07:46:30+00:00",
15 − "data_hashes": {},
15 + "data_hashes": {
16 + "addthree": {
17 + "hash": "e2d74cd2fbf01431",
18 + "n_queries": 35
19 + },
20 + "antonym": {
21 + "hash": "6f74ac11f9505135",
22 + "n_queries": 32
23 + },
24 + "capital": {
25 + "hash": "78ba5b39d43d42d5",
26 + "n_queries": 22
27 + },
28 + "morph": {
29 + "hash": "df15df1dd69d1028",
30 + "n_queries": 40
31 + }
32 + },
16 33 "experiment_id": "tier1_taskvector",
17 34 "git": {
18 35 "branch": "main",
@@ -34,6 +51,11 @@
34 51 "note": "",
35 52 "status": "running",
36 53 "utc": "2026-11-09T07:46:33+00:00"
54 + },
55 + {
56 + "note": "",
57 + "status": "completed",
58 + "utc": "2026-11-09T08:14:02+00:00"
37 59 }
38 60 ],
39 61 "interpretation": "",
@@ -48,9 +70,11 @@
48 70 "torch": "2.12.0",
49 71 "transformers": "5.8.1"
50 72 },
51 − "outputs": {},
73 + "outputs": {
74 + "results": "experiments/summaries/tier1_taskvector_e105e3d91386/results.json"
75 + },
52 76 "run_id": "tier1_taskvector_e105e3d91386",
53 77 "seeds": {},
54 − "status": "running",
78 + "status": "completed",
55 79 "warnings": []
56 80 }
\ No newline at end of file
added experiments/manifests/tier1_taskvector_eb5d091bf5ed.json +77 −0
@@ -0,0 +1,77 @@
1 +{
2 + "config": {
3 + "k": 8,
4 + "pairs": [
5 + "qwen7b:qwen3b",
6 + "qwen3b:qwen7b",
7 + "qwen7b:llama3b"
8 + ],
9 + "quick": false,
10 + "split": "dev",
11 + "task_seed": 0
12 + },
13 + "created_utc": "2026-11-09T07:54:32+00:00",
14 + "data_hashes": {
15 + "addthree": {
16 + "hash": "e2d74cd2fbf01431",
17 + "n_queries": 35
18 + },
19 + "antonym": {
20 + "hash": "6f74ac11f9505135",
21 + "n_queries": 32
22 + },
23 + "capital": {
24 + "hash": "78ba5b39d43d42d5",
25 + "n_queries": 22
26 + },
27 + "morph": {
28 + "hash": "df15df1dd69d1028",
29 + "n_queries": 40
30 + }
31 + },
32 + "experiment_id": "tier1_taskvector",
33 + "git": {
34 + "branch": null,
35 + "commit": null,
36 + "dirty": null
37 + },
38 + "hardware": {
39 + "chip": "Apple M3 Ultra",
40 + "cpu_count": 28,
41 + "hostname_hash": "1f59c4ad75ba",
42 + "machine": "arm64",
43 + "memory_gb": 96.0,
44 + "platform": "macOS-26.4-arm64-arm-64bit-Mach-O"
45 + },
46 + "history": [
47 + {
48 + "note": "",
49 + "status": "running",
50 + "utc": "2026-11-09T07:54:32+00:00"
51 + },
52 + {
53 + "note": "",
54 + "status": "completed",
55 + "utc": "2026-11-09T08:36:41+00:00"
56 + }
57 + ],
58 + "interpretation": "",
59 + "libraries": {
60 + "mlx": "?",
61 + "mlx_lm": "0.31.3",
62 + "numpy": "2.4.4",
63 + "pydantic": "2.13.4",
64 + "python": "3.14.4",
65 + "safetensors": "0.7.0",
66 + "scipy": "1.17.1",
67 + "torch": null,
68 + "transformers": "5.10.2"
69 + },
70 + "outputs": {
71 + "results": "experiments/summaries/tier1_taskvector_eb5d091bf5ed/results.json"
72 + },
73 + "run_id": "tier1_taskvector_eb5d091bf5ed",
74 + "seeds": {},
75 + "status": "completed",
76 + "warnings": []
77 +}
\ No newline at end of file
modified experiments/registry/runs.jsonl +12 −0
@@ -23,3 +23,15 @@
23 23 {"experiment_id": "graft_tier0_confirmatory", "results": "experiments/summaries/graft_tier0_confirmatory_0c2257f6803e/results.json", "run_id": "graft_tier0_confirmatory_0c2257f6803e", "status": "completed", "utc": "2026-11-09T07:44:03+00:00"}
24 24 {"error": "ValueError(\"invalid literal for int() with base 10: 'rocrustes'\")", "experiment_id": "tier1_taskvector", "run_id": "tier1_taskvector_e105e3d91386", "status": "failed", "utc": "2026-11-09T07:45:09+00:00"}
25 25 {"config_hash": "e105e3d91386", "experiment_id": "tier1_taskvector", "run_id": "tier1_taskvector_e105e3d91386", "status": "planned", "utc": "2026-11-09T07:46:33+00:00"}
26 +{"experiment_id": "tier1_taskvector", "run_id": "tier1_taskvector_e105e3d91386", "status": "completed", "utc": "2026-11-09T08:14:02+00:00"}
27 +{"config_hash": "82a78041c0ce", "experiment_id": "tier1_taskvector_confirmatory", "run_id": "tier1_taskvector_confirmatory_82a78041c0ce", "status": "planned", "utc": "2026-11-09T08:15:06+00:00"}
28 +{"config_hash": "eb5d091bf5ed", "experiment_id": "tier1_taskvector", "run_id": "tier1_taskvector_eb5d091bf5ed", "status": "planned", "utc": "2026-11-09T07:50:22+00:00"}
29 +{"error": "ModuleNotFoundError(\"No module named 'torch'\")", "experiment_id": "tier1_taskvector", "run_id": "tier1_taskvector_eb5d091bf5ed", "status": "failed", "utc": "2026-11-09T07:50:22+00:00"}
30 +{"config_hash": "eb5d091bf5ed", "experiment_id": "tier1_taskvector", "run_id": "tier1_taskvector_eb5d091bf5ed", "status": "planned", "utc": "2026-11-09T07:51:06+00:00"}
31 +{"config_hash": "eb5d091bf5ed", "experiment_id": "tier1_taskvector", "run_id": "tier1_taskvector_eb5d091bf5ed", "status": "planned", "utc": "2026-11-09T07:54:32+00:00"}
32 +{"experiment_id": "tier1_taskvector", "run_id": "tier1_taskvector_eb5d091bf5ed", "status": "completed", "utc": "2026-11-09T08:36:41+00:00"}
33 +{"experiment_id": "tier1_taskvector_confirmatory", "run_id": "tier1_taskvector_confirmatory_82a78041c0ce", "status": "completed", "utc": "2026-11-09T08:41:48+00:00"}
34 +{"utc": "2026-11-09T08:50:00+00:00", "run_id": "tier1_taskvector_confirmatory_82a78041c0ce", "experiment_id": "tier1_taskvector_confirmatory", "status": "invalidated", "note": "layers not frozen from dev (A2/D13)"}
35 +{"config_hash": "82a78041c0ce", "experiment_id": "tier1_taskvector_confirmatory", "run_id": "tier1_taskvector_confirmatory_82a78041c0ce", "status": "planned", "utc": "2026-11-09T08:42:58+00:00"}
36 +{"config_hash": "625c54bd186f", "experiment_id": "tier1_taskvector_confirmatory", "run_id": "tier1_taskvector_confirmatory_625c54bd186f", "status": "planned", "utc": "2026-11-09T08:44:08+00:00"}
37 +{"experiment_id": "tier1_taskvector_confirmatory", "run_id": "tier1_taskvector_confirmatory_625c54bd186f", "status": "completed", "utc": "2026-11-09T09:06:53+00:00"}
added experiments/summaries/tier1_taskvector_confirmatory_625c54bd186f/TIER1_CONFIRMATORY.md +26 −0
@@ -0,0 +1,26 @@
1 +# Tier-1/2 confirmatory analysis — dev `tier1_taskvector_e105e3d91386`, confirmatory `tier1_taskvector_confirmatory_625c54bd186f`
2 +
3 +| pair | task | eligible (own vector dev) | selected condition (dev acc) | conf: zero / few / own / selected / best control | selected − best control [95% CI] |
4 +|---|---|---|---|---|---|
5 +| qwen3b->qwen1.5b | antonym | yes (0.88) | replace_donor_fewshot|procrustes|L16 (0.72) | 0.19 / 0.94 / 0.91 / **0.59** / 0.44 (random_orth|L19|a1.0) | +0.16 [+0.00, +0.34]; pre-A1 control 0.69; shuffled-query diag. 0.69 |
6 +| qwen3b->qwen1.5b | capital | yes (0.86) | ridge|L22|a1.0 (0.18) | 0.00 / 1.00 / 0.82 / **0.09** / 0.00 (random_orth|L19|a1.0) | +0.09 [+0.00, +0.23]; pre-A1 control 0.00; shuffled-query diag. 0.00 |
7 +| qwen3b->qwen1.5b | morph | no (0.00) | procrustes|L2|a1.0 (0.00) | 0.00 / 0.30 / 0.00 / **0.00** / 0.00 (random_orth|L2|a1.0) | +0.00 [+0.00, +0.00]; pre-A1 control 0.00; shuffled-query diag. 0.00 |
8 +| qwen3b->qwen1.5b | addthree | no (0.00) | procrustes|L2|a1.0 (0.00) | 0.00 / 0.86 / 0.00 / **0.00** / 0.00 (random_orth|L2|a1.0) | +0.00 [+0.00, +0.00]; pre-A1 control 0.00; shuffled-query diag. 0.00 |
9 +| **qwen3b->qwen1.5b** | | | | | **H6 not supported (0 task(s))** |
10 +| qwen1.5b->qwen3b | antonym | yes (0.88) | procrustes_rescaled|L21|a1.0 (0.75) | 0.09 / 0.94 / 0.91 / **0.69** / 0.41 (ctl_unrelated_task_vector(capital)|L25) | +0.28 [+0.09, +0.47]; pre-A1 control 0.81; shuffled-query diag. 0.81 |
11 +| qwen1.5b->qwen3b | capital | yes (0.82) | procrustes|L28|a1.0 (0.73) | 0.14 / 1.00 / 0.82 / **0.55** / 0.18 (ctl_unrelated_task_vector(antonym)|L28) | +0.36 [+0.14, +0.59] ✔; pre-A1 control 0.27; shuffled-query diag. 0.27 |
12 +| qwen1.5b->qwen3b | morph | no (0.00) | procrustes|L2|a1.0 (0.00) | 0.00 / 0.40 / 0.00 / **0.00** / 0.00 (random_orth|L2|a1.0) | +0.00 [+0.00, +0.00]; pre-A1 control 0.00; shuffled-query diag. 0.00 |
13 +| qwen1.5b->qwen3b | addthree | no (0.00) | procrustes|L2|a1.0 (0.00) | 0.00 / 1.00 / 0.00 / **0.00** / 0.00 (random_orth|L2|a1.0) | +0.00 [+0.00, +0.00]; pre-A1 control 0.00; shuffled-query diag. 0.00 |
14 +| **qwen1.5b->qwen3b** | | | | | **H6 partial (1 task(s))** |
15 +| llama3b->qwen3b | antonym | yes (0.88) | cca|L25|a1.0 (0.81) | 0.09 / 0.94 / 0.91 / **0.81** / 0.31 (ctl_unrelated_task_vector(capital)|L25) | +0.50 [+0.31, +0.69] ✔; pre-A1 control 0.81; shuffled-query diag. 0.81 |
16 +| llama3b->qwen3b | capital | yes (0.82) | cca|L18|a1.0 (0.50) | 0.14 / 1.00 / 0.82 / **0.64** / 0.45 (ctl_replace_donor_zeroshot|sham_map|L18) | +0.18 [+0.05, +0.36]; pre-A1 control 0.45; shuffled-query diag. 0.41 |
17 +| llama3b->qwen3b | morph | no (0.00) | procrustes|L2|a1.0 (0.00) | 0.00 / 0.40 / 0.00 / **0.00** / 0.00 (random_orth|L2|a1.0) | +0.00 [+0.00, +0.00]; pre-A1 control 0.00; shuffled-query diag. 0.00 |
18 +| llama3b->qwen3b | addthree | no (0.00) | procrustes|L2|a1.0 (0.00) | 0.00 / 1.00 / 0.00 / **0.00** / 0.00 (random_orth|L2|a1.0) | +0.00 [+0.00, +0.00]; pre-A1 control 0.00; shuffled-query diag. 0.00 |
19 +| **llama3b->qwen3b** | | | | | **H6 partial (1 task(s))** |
20 +| qwen3b->llama3b | antonym | yes (0.84) | procrustes|L14|a1.0 (0.66) | 0.06 / 0.91 / 0.84 / **0.59** / 0.34 (ctl_unrelated_task_vector(capital)|L19) | +0.25 [+0.09, +0.41]; pre-A1 control 0.38; shuffled-query diag. 0.38 |
21 +| qwen3b->llama3b | capital | yes (1.00) | procrustes|L14|a1.0 (0.05) | 0.09 / 1.00 / 1.00 / **0.09** / 0.14 (ctl_unrelated_task_vector(antonym)|L14) | -0.05 [-0.18, +0.09]; pre-A1 control 0.14; shuffled-query diag. 0.09 |
22 +| qwen3b->llama3b | morph | no (0.00) | procrustes|L2|a1.0 (0.00) | 0.00 / 0.28 / 0.00 / **0.00** / 0.00 (random_orth|L2|a1.0) | +0.00 [+0.00, +0.00]; pre-A1 control 0.00; shuffled-query diag. 0.00 |
23 +| qwen3b->llama3b | addthree | yes (0.80) | procrustes|L2|a1.0 (0.00) | 0.00 / 1.00 / 0.69 / **0.00** / 0.00 (random_orth|L2|a1.0) | +0.00 [+0.00, +0.00]; pre-A1 control 0.00; shuffled-query diag. 0.00 |
24 +| **qwen3b->llama3b** | | | | | **H6 not supported (0 task(s))** |
25 +
26 +Note: CIs are exact paired item bootstraps (2 000 draws) on per-item correctness stored in the confirmatory results.
added experiments/summaries/tier1_taskvector_confirmatory_625c54bd186f/results.json +19597 −0
Large diff (19,597 lines) is not loaded by default to keep the page fast.Load diff

Diff truncated — file too large.

Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere