SPB Git
spboucher.ai

spb/wp29_uqo

Public

Task suite (F1,F2,F4,F5,F6 + partial F3), evaluator validation, analysis pipeline, paper skeleton, pilot batch 1 launched

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Oct 30, 2026) parent 76de75e Browse files

375 changed files +19,996 −0

375 changed files
Load all 375 files Unified Split
added README.md +38 −0
@@ -0,0 +1,38 @@
1 +<!-- Author: Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca -->
2 +# Protocol-Guided Autonomous Research with Claude Code
3 +
4 +**UQO Working Paper No. 29** — *Scientific Verification, Reproducibility, and the Cost of Delegation: A Controlled Study*
5 +
6 +> A controlled, four-arm experiment in which fresh, isolated Claude Code sessions (bare mode, `claude-opus-5-5`) solve bounded scientific tasks under (A) an open instruction, (B) a structured research protocol, (C) the protocol plus controller-enforced executable gates, and (D) the same plus a separate review-and-repair session. Outcomes are scored by deterministic hidden oracles on a separate host. Every number in the paper is regenerated from `runs/state.sqlite`.
7 +
8 +## Status
9 +See `reports/phase_status.md` (append-only phase log), `protocol/amendments.csv` and `reports/interventions.csv`. The complete research mandate is `protocol/WP29_FULL_MANDATE.md`.
10 +
11 +## Layout
12 +```
13 +protocol/ mandate, preregistration, contract, treatments (arms/), evaluation plan, screenshot protocol, amendments
14 +research/ corpus of 11 source mandates (hashed), coding, literature matrix + verified BibTeX
15 +configs/ budgets and per-split node/envelope configuration
16 +tasks/ TASK_AUTHORING_SPEC.md, registry.csv, task_cards/, public_bundles/ (hidden oracles live in ~/wp29-oracles/, outside the repo)
17 +src/pgar/ harness: contracts, state (SQLite), events, launcher (ssh + claude --bare), isolation, budgets, runtime_gates (gate program), recovery, scheduler, evaluation, exports, capture, evidence_viewer, cli
18 +evaluators/ public receipt schema + run_validations.py (seeded evaluator validation)
19 +tests/ infrastructure tests (no provider access)
20 +runs/ state.sqlite + per-run directories (raw streams are not committed)
21 +evidence/ screenshot originals (restricted, not committed) and public copies
22 +analysis/ build_analysis_dataset, primary_analysis, robustness, figures, tables
23 +results/ analysis_data (CSV exports), estimates, figures (PDF), tables (LaTeX)
24 +paper/ LaTeX manuscript (UQO template)
25 +reports/ environment/isolation audits, evaluator validation, pilot report, accounting, final review
26 +```
27 +
28 +## Commands (laptop = controller; see Makefile)
29 +```bash
30 +make test # infrastructure tests (state machine, gate, evaluator, bundle hygiene)
31 +make deploy # install the gate program + key file on executor nodes
32 +make probe # sentinel isolation probes on the nodes (tiny live calls)
33 +make smoke-live # 1 task × 4 arms, live
34 +PYTHONPATH=src .venv/bin/python -m pgar.cli assign --split pilot --tasks <ids> --repeats 1 --seed 202
35 +PYTHONPATH=src .venv/bin/python -m pgar.cli run --split pilot --workers 8
36 +make analyze && make paper
37 +```
38 +Requirements: Python 3.12 venv (`uv venv .venv --python 3.12 && uv pip install -p .venv/bin/python -e .`), SSH access to the executor nodes, `ANTHROPIC_API_KEY` in `~/wp29/.env` on each node, Claude Code 2.1.314 on the nodes.
added analysis/build_analysis_dataset.py +37 −0
@@ -0,0 +1,37 @@
1 +"""Build the frozen analysis dataset from the state store (exports + data-quality assertions + negative controls)."""
2 +import os, sys, json, hashlib, datetime
3 +sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src")); sys.path.insert(0, os.path.dirname(__file__))
4 +from pgar.state import StateStore
5 +from pgar.exports import export_runs
6 +from common import REPO, DATA, load_runs
7 +import pandas as pd
8 +
9 +def main():
10 + store = StateStore(os.path.join(REPO, "runs", "state.sqlite"))
11 + export_runs(store, REPO)
12 + df = pd.read_csv(os.path.join(DATA, "runs.csv"))
13 + # data-quality assertions
14 + problems = []
15 + for split in df["split"].unique():
16 + d = df[df["split"] == split]
17 + counts = d.groupby(["task_id", "treatment"]).size().unstack(fill_value=0)
18 + if split in ("pilot", "confirmatory") and (counts.nunique(axis=1) > 1).any():
19 + problems.append(f"{split}: unbalanced arms per task\n{counts[counts.nunique(axis=1) > 1]}")
20 + dup = d.duplicated(subset=["task_id", "treatment", "repeat_index"]).sum()
21 + if dup: problems.append(f"{split}: {dup} duplicate (task, arm, repeat) cells")
22 + ev = df[df["status"] == "evaluated"]
23 + if (ev["scientific_success"].isna()).any(): problems.append("evaluated runs without scientific_success")
24 + if (ev["spent_usd"].fillna(0) < 0).any(): problems.append("negative cost")
25 + # negative control: Y must be 0 whenever n_violations>0 or any criterion failed
26 + bad = ev[(ev["scientific_success"] == 1) & ((ev["n_violations"] > 0) | (ev["n_criteria_passed"] < ev["n_criteria"]))]
27 + if len(bad): problems.append(f"{len(bad)} runs with success=1 despite violations/failed criteria: {bad['run_id'].tolist()}")
28 + summary = {"built_utc": datetime.datetime.now(datetime.timezone.utc).isoformat(), "n_runs": int(len(df)), "by_split_status": df.groupby(["split", "status"]).size().to_dict(),
29 + "problems": problems, "runs_csv_sha256": hashlib.sha256(open(os.path.join(DATA, "runs.csv"), "rb").read()).hexdigest()}
30 + summary["by_split_status"] = {f"{k[0]}/{k[1]}": int(v) for k, v in summary["by_split_status"].items()}
31 + json.dump(summary, open(os.path.join(DATA, "build_summary.json"), "w"), indent=1)
32 + print(json.dumps(summary, indent=1))
33 + if problems:
34 + print("DATA-QUALITY PROBLEMS:", *problems, sep="\n", file=sys.stderr)
35 +
36 +if __name__ == "__main__":
37 + main()
added analysis/common.py +47 −0
@@ -0,0 +1,47 @@
1 +"""Shared helpers for the WP29 analysis pipeline (frozen interfaces)."""
2 +import os, json, numpy as np, pandas as pd
3 +REPO = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
4 +DATA = os.path.join(REPO, "results", "analysis_data"); EST = os.path.join(REPO, "results", "estimates")
5 +ARMS = ["A", "B", "C", "D"]
6 +
7 +def load_runs(split=None, path=None):
8 + df = pd.read_csv(path or os.path.join(DATA, "runs.csv"))
9 + if split: df = df[df["split"] == split]
10 + df = df.copy()
11 + df["Y"] = ((df["scientific_success"] == 1) & (df["status"].isin(["evaluated"])) & (df["terminal_reason"].fillna("").str.startswith("executor_finished") | df["terminal_reason"].fillna("").str.startswith("result_") | df["terminal_reason"].fillna("").eq("max_turns_reached"))).astype(int)
12 + # Y requires completion within the envelope: timeout/budget_exhausted → 0 (status is 'evaluated' for all evaluated runs; terminal_reason keeps the cause)
13 + df.loc[df["terminal_reason"].fillna("").isin(["wall_clock_limit"]) | df["terminal_reason"].fillna("").str.contains("budget", na=False), "Y"] = 0
14 + df["Y_any"] = (df["scientific_success"] == 1).astype(int) # correct regardless of envelope
15 + df["viol"] = (df["n_violations"].fillna(0) > 0).astype(int)
16 + df["family"] = df["task_id"].str.split("-").str[0]
17 + return df
18 +
19 +def task_weighted_rate(df, arm, col="Y"):
20 + d = df[df["treatment"] == arm]
21 + if d.empty: return np.nan
22 + return d.groupby("task_id")[col].mean().mean()
23 +
24 +def paired_task_diff(df, a, b, col="Y"):
25 + """Task-level paired differences (mean over repeats within task) a − b, as a Series indexed by task."""
26 + ma = df[df["treatment"] == a].groupby("task_id")[col].mean(); mb = df[df["treatment"] == b].groupby("task_id")[col].mean()
27 + common = ma.index.intersection(mb.index)
28 + return (ma.loc[common] - mb.loc[common])
29 +
30 +def cluster_bootstrap(diffs, B=5000, seed=20261030):
31 + rng = np.random.default_rng(seed); x = diffs.values; n = len(x)
32 + if n == 0: return (np.nan, np.nan, np.nan)
33 + boots = np.array([x[rng.integers(0, n, n)].mean() for _ in range(B)])
34 + return float(x.mean()), float(np.percentile(boots, 2.5)), float(np.percentile(boots, 97.5))
35 +
36 +def permutation_p(diffs, n_perm=20000, seed=20261030):
37 + rng = np.random.default_rng(seed); x = diffs.values; n = len(x)
38 + if n == 0: return np.nan
39 + obs = abs(x.mean()); signs = rng.choice([-1, 1], size=(n_perm, n))
40 + perm = np.abs((signs * x).mean(axis=1))
41 + return float((np.sum(perm >= obs - 1e-12) + 1) / (n_perm + 1))
42 +
43 +def holm(pvals: dict):
44 + items = sorted(pvals.items(), key=lambda kv: kv[1]); m = len(items); out = {}; prev = 0.0
45 + for i, (k, p) in enumerate(items):
46 + adj = min(1.0, max(prev, (m - i) * p)); out[k] = adj; prev = adj
47 + return out
added analysis/figures.py +91 −0
@@ -0,0 +1,91 @@
1 +"""Quantitative figures (vector PDF) generated from the frozen analysis dataset. Every figure shows n and uncertainty where relevant."""
2 +import os, sys, json
3 +sys.path.insert(0, os.path.dirname(__file__))
4 +from common import *
5 +import matplotlib; matplotlib.use("Agg")
6 +import matplotlib.pyplot as plt
7 +FIG = os.path.join(REPO, "results", "figures"); os.makedirs(FIG, exist_ok=True)
8 +COL = {"A": "#7f7f7f", "B": "#4c72b0", "C": "#dd8452", "D": "#55a868"}
9 +plt.rcParams.update({"font.size": 9, "axes.spines.top": False, "axes.spines.right": False, "pdf.fonttype": 42})
10 +
11 +def fig_primary(split):
12 + at = pd.read_csv(os.path.join(EST, f"{split}_arm_table.csv")); ct = pd.read_csv(os.path.join(EST, f"{split}_contrasts.csv")); ct = ct[ct["endpoint"] == "Y"]
13 + df = load_runs(split)
14 + fig, ax = plt.subplots(1, 2, figsize=(7.4, 3.1), gridspec_kw={"width_ratios": [1, 1.4]})
15 + # left: task-weighted rates with cluster-bootstrap CI (per arm)
16 + for i, a in enumerate(ARMS):
17 + d = df[df["treatment"] == a].groupby("task_id")["Y"].mean()
18 + est, lo, hi = cluster_bootstrap(d - 0, 3000)
19 + ax[0].bar(i, est, color=COL[a], width=0.65); ax[0].errorbar(i, est, yerr=[[est - lo], [hi - est]], color="k", capsize=3, lw=1)
20 + ax[0].text(i, min(hi + 0.03, 1.02), f"{len(d)} tasks\n{len(df[df['treatment']==a])} runs", ha="center", va="bottom", fontsize=6.5)
21 + ax[0].set_xticks(range(4)); ax[0].set_xticklabels([f"Arm {a}" for a in ARMS]); ax[0].set_ylim(0, 1.2); ax[0].set_ylabel("Verified completion $\\hat p_a$ (task-weighted)")
22 + ax[0].set_title("(a) Primary endpoint by arm", loc="left")
23 + # right: paired contrasts
24 + cts = ct.reset_index(drop=True)
25 + for i, r in cts.iterrows():
26 + ax[1].errorbar(r["estimate"], i, xerr=[[r["estimate"] - r["ci_low"]], [r["ci_high"] - r["estimate"]]], fmt="o", color="k" if r["role"] == "primary" else "#555", capsize=3)
27 + ax[1].text(0.99, i + 0.32, f"p={r['perm_p']:.3f}; tasks +/=/−: {int(r['n_tasks_favoring_first'])}/{int(r['n_tasks_tied'])}/{int(r['n_tasks_favoring_second'])}", fontsize=6, va="center", ha="right", transform=ax[1].get_yaxis_transform(), color="#444")
28 + ax[1].axvline(0, color="#999", lw=0.8); ax[1].set_yticks(range(len(cts))); ax[1].set_yticklabels([f"{c} ({r})" for c, r in zip(cts["contrast"], cts["role"])]); ax[1].invert_yaxis()
29 + ax[1].set_xlabel("Paired task-level difference (95% task-cluster bootstrap CI)"); ax[1].set_title("(b) Prespecified contrasts", loc="left")
30 + fig.tight_layout(); fig.savefig(os.path.join(FIG, f"fig_primary_{split}.pdf")); plt.close(fig)
31 +
32 +def fig_quality(split):
33 + df = load_runs(split)
34 + comps = ["q_correctness", "q_integrity", "q_reproducibility", "q_reporting", "q_baseline"]; labels = ["correctness", "integrity", "reproducibility", "reporting", "baseline"]
35 + fig, ax = plt.subplots(1, 2, figsize=(7.2, 2.8))
36 + w = 0.2
37 + for i, a in enumerate(ARMS):
38 + d = df[df["treatment"] == a]; means = [d.groupby("task_id")[c].mean().mean() for c in comps]
39 + ax[0].bar(np.arange(len(comps)) + (i - 1.5) * w, means, width=w, color=COL[a], label=f"Arm {a} (n={len(d)})")
40 + ax[0].set_xticks(range(len(comps))); ax[0].set_xticklabels(labels, rotation=20); ax[0].set_ylim(0, 1.05); ax[0].set_ylabel("Mean component score (task-weighted)"); ax[0].legend(fontsize=6, frameon=False); ax[0].set_title("(a) Quality components", loc="left")
41 + # violations by type and arm
42 + cr = pd.read_csv(os.path.join(DATA, "criteria.csv")); cr = cr[cr["kind"] == "violation"].merge(df[["run_id", "treatment"]], on="run_id")
43 + if len(cr):
44 + tab = cr.groupby(["passed_or_type", "treatment"])["run_id"].nunique().unstack(fill_value=0).reindex(columns=ARMS, fill_value=0)
45 + n_runs = df.groupby("treatment").size().reindex(ARMS)
46 + rate = tab / n_runs
47 + rate.plot.bar(ax=ax[1], color=[COL[a] for a in ARMS], width=0.8); ax[1].set_ylabel("Share of runs with ≥1 violation of type"); ax[1].set_xlabel(""); ax[1].legend(fontsize=6, frameon=False, title=None)
48 + ax[1].set_xticklabels(ax[1].get_xticklabels(), rotation=20)
49 + ax[1].set_title("(b) Critical integrity violations by type", loc="left")
50 + fig.tight_layout(); fig.savefig(os.path.join(FIG, f"fig_quality_{split}.pdf")); plt.close(fig)
51 +
52 +def fig_cost(split):
53 + df = load_runs(split)
54 + fig, ax = plt.subplots(1, 2, figsize=(7.2, 2.8))
55 + for a in ARMS:
56 + d = df[df["treatment"] == a]
57 + ax[0].scatter(d["spent_usd"], d["spent_wall_s"] / 60, s=14, color=COL[a], alpha=0.7, label=f"Arm {a}", marker="o" if True else "x")
58 + ax[1].scatter(d["spent_usd"].mean(), task_weighted_rate(d, a), s=60, color=COL[a], edgecolor="k", zorder=3)
59 + ax[1].annotate(f"{a}", (d["spent_usd"].mean(), task_weighted_rate(d, a)), xytext=(5, 4), textcoords="offset points", fontsize=8)
60 + ax[1].errorbar(d["spent_usd"].mean(), task_weighted_rate(d, a), xerr=d["spent_usd"].std() / np.sqrt(max(1, len(d))), color=COL[a], capsize=2, lw=0.8)
61 + ax[0].set_xlabel("Inference cost per run (USD, client estimate)"); ax[0].set_ylabel("Wall time per run (min)"); ax[0].legend(fontsize=6, frameon=False); ax[0].set_title("(a) Cost and time of every run", loc="left")
62 + ax[1].set_xlabel("Mean cost per run (USD) ± s.e."); ax[1].set_ylabel("Intervention-free verified completion"); ax[1].set_ylim(0, 1.05); ax[1].set_title("(b) Quality–cost frontier", loc="left")
63 + fig.tight_layout(); fig.savefig(os.path.join(FIG, f"fig_cost_{split}.pdf")); plt.close(fig)
64 +
65 +def fig_family(split):
66 + fam = pd.read_csv(os.path.join(EST, f"{split}_family_table.csv"))
67 + fams = sorted(fam["family"].unique()); fig, ax = plt.subplots(figsize=(7.2, 2.6)); w = 0.2
68 + for i, a in enumerate(ARMS):
69 + d = fam[fam["treatment"] == a].set_index("family").reindex(fams)
70 + ax.bar(np.arange(len(fams)) + (i - 1.5) * w, d["Y"], width=w, color=COL[a], label=f"Arm {a}")
71 + for j, f in enumerate(fams):
72 + if i == 0: ax.text(j, 1.02, f"{int(d.loc[f, 'n_tasks'])} tasks", ha="center", fontsize=7)
73 + ax.set_xticks(range(len(fams))); ax.set_xticklabels(fams); ax.set_ylim(0, 1.12); ax.set_ylabel("Completion (task-weighted)"); ax.legend(fontsize=7, frameon=False, ncol=4); ax.set_title("Task-family heterogeneity of the primary endpoint", loc="left")
74 + fig.tight_layout(); fig.savefig(os.path.join(FIG, f"fig_family_{split}.pdf")); plt.close(fig)
75 +
76 +def fig_errors(split):
77 + df = load_runs(split); cr = pd.read_csv(os.path.join(DATA, "criteria.csv")).merge(df[["run_id", "treatment", "task_id"]], on="run_id")
78 + fails = cr[(cr["kind"] == "criterion") & (cr["passed_or_type"].astype(str) == "0")]
79 + if fails.empty: return
80 + fails["error"] = fails["task_id"] + ":" + fails["id"]
81 + top = fails.groupby("error")["run_id"].nunique().sort_values(ascending=False).head(25)
82 + tab = fails[fails["error"].isin(top.index)].groupby(["error", "treatment"])["run_id"].nunique().unstack(fill_value=0).reindex(columns=ARMS, fill_value=0).loc[top.index]
83 + fig, ax = plt.subplots(figsize=(7.2, 0.22 * len(tab) + 1.2)); tab.plot.barh(ax=ax, stacked=True, color=[COL[a] for a in ARMS], width=0.8)
84 + ax.invert_yaxis(); ax.set_xlabel("Number of runs failing the criterion (denominator: runs of that task per arm)"); ax.legend(fontsize=7, frameon=False); ax.set_title("Most frequent failed scientific criteria (task:criterion)", loc="left")
85 + fig.tight_layout(); fig.savefig(os.path.join(FIG, f"fig_errors_{split}.pdf")); plt.close(fig)
86 +
87 +if __name__ == "__main__":
88 + split = sys.argv[1] if len(sys.argv) > 1 else "confirmatory"
89 + for f in (fig_primary, fig_quality, fig_cost, fig_family, fig_errors):
90 + try: f(split); print("ok", f.__name__)
91 + except Exception as e: print("skip", f.__name__, e)
added analysis/primary_analysis.py +50 −0
@@ -0,0 +1,50 @@
1 +"""Primary analysis: task-weighted completion by arm, prespecified contrasts (D−A primary; B−A, C−B, D−C secondary with Holm),
2 +task-cluster bootstrap CIs and paired sign-flip permutation p-values. Also secondary endpoints by arm."""
3 +import os, sys, json
4 +sys.path.insert(0, os.path.dirname(__file__))
5 +from common import *
6 +
7 +def arm_table(df):
8 + rows = []
9 + for a in ARMS:
10 + d = df[df["treatment"] == a]
11 + rows.append({"arm": a, "n_runs": int(len(d)), "n_tasks": int(d["task_id"].nunique()), "n_evaluated": int((d["status"] == "evaluated").sum()),
12 + "Y_task_weighted": task_weighted_rate(d, a, "Y"), "Y_run_mean": float(d["Y"].mean()) if len(d) else np.nan,
13 + "correct_any_task_weighted": task_weighted_rate(d, a, "Y_any"), "violation_rate": float(d["viol"].mean()) if len(d) else np.nan,
14 + "n_runs_with_violation": int(d["viol"].sum()), "reproducible_rate": float(d["reproducibility"].mean()) if len(d) else np.nan,
15 + "quality_mean": float(d["quality_score"].mean()) if len(d) else np.nan, "q_correctness": float(d["q_correctness"].mean()) if len(d) else np.nan,
16 + "q_reporting": float(d["q_reporting"].mean()) if len(d) else np.nan, "q_baseline": float(d["q_baseline"].mean()) if len(d) else np.nan,
17 + "cost_mean_usd": float(d["spent_usd"].mean()) if len(d) else np.nan, "cost_median_usd": float(d["spent_usd"].median()) if len(d) else np.nan,
18 + "cost_p90_usd": float(d["spent_usd"].quantile(0.9)) if len(d) else np.nan, "cost_total_usd": float(d["spent_usd"].sum()),
19 + "reviewer_usd_total": float(d["reviewer_usd"].fillna(0).sum()), "wall_median_s": float(d["spent_wall_s"].median()) if len(d) else np.nan,
20 + "wall_p90_s": float(d["spent_wall_s"].quantile(0.9)) if len(d) else np.nan, "n_timeout": int((d["terminal_reason"] == "wall_clock_limit").sum()),
21 + "n_budget_exhausted": int(d["terminal_reason"].fillna("").str.contains("budget").sum()), "n_sessions_mean": float(d["n_sessions"].mean()) if len(d) else np.nan,
22 + "cost_per_success_usd": (float(d["spent_usd"].sum() / d["Y"].sum()) if d["Y"].sum() > 0 else None)})
23 + return pd.DataFrame(rows)
24 +
25 +def contrasts(df, col="Y", B=5000, n_perm=20000):
26 + out = []
27 + pv = {}
28 + for (a, b, role) in [("D", "A", "primary"), ("B", "A", "secondary"), ("C", "B", "secondary"), ("D", "C", "secondary"), ("C", "A", "exploratory"), ("D", "B", "exploratory")]:
29 + d = paired_task_diff(df, a, b, col); est, lo, hi = cluster_bootstrap(d, B); p = permutation_p(d, n_perm)
30 + out.append({"contrast": f"{a}-{b}", "role": role, "endpoint": col, "n_task_clusters": int(len(d)), "estimate": est, "ci_low": lo, "ci_high": hi, "perm_p": p,
31 + "n_tasks_favoring_first": int((d > 0).sum()), "n_tasks_favoring_second": int((d < 0).sum()), "n_tasks_tied": int((d == 0).sum())})
32 + if role == "secondary": pv[f"{a}-{b}"] = p
33 + adj = holm(pv) if pv else {}
34 + for r in out:
35 + r["holm_p"] = adj.get(r["contrast"])
36 + return pd.DataFrame(out)
37 +
38 +def main(split="confirmatory"):
39 + os.makedirs(EST, exist_ok=True)
40 + df = load_runs(split)
41 + if df.empty:
42 + print(f"no runs for split {split}"); return
43 + at = arm_table(df); at.to_csv(os.path.join(EST, f"{split}_arm_table.csv"), index=False)
44 + ct = pd.concat([contrasts(df, "Y"), contrasts(df, "Y_any"), contrasts(df, "viol"), contrasts(df, "quality_score")]); ct.to_csv(os.path.join(EST, f"{split}_contrasts.csv"), index=False)
45 + fam = df.groupby(["family", "treatment"]).apply(lambda d: pd.Series({"Y": d.groupby("task_id")["Y"].mean().mean(), "n_tasks": d["task_id"].nunique(), "n_runs": len(d), "viol": d["viol"].mean(), "cost_mean": d["spent_usd"].mean()}), include_groups=False).reset_index()
46 + fam.to_csv(os.path.join(EST, f"{split}_family_table.csv"), index=False)
47 + print(at.round(3).to_string()); print(); print(ct.round(4).to_string())
48 +
49 +if __name__ == "__main__":
50 + main(sys.argv[1] if len(sys.argv) > 1 else "confirmatory")
added analysis/robustness.py +27 −0
@@ -0,0 +1,27 @@
1 +"""Robustness: alternative endpoint definitions, leave-one-family-out, run-level (non task-weighted) aggregation, timeout handling,
2 +block-order effects (early vs late in block), node effects, sensitivity to uncertain rubric labels (tasks flagged uncertain in the registry notes)."""
3 +import os, sys, json
4 +sys.path.insert(0, os.path.dirname(__file__))
5 +from common import *
6 +
7 +def main(split="confirmatory"):
8 + df = load_runs(split)
9 + if df.empty: print("no runs"); return
10 + rows = []
11 + def add(name, d, col="Y"):
12 + dd = paired_task_diff(d, "D", "A", col); est, lo, hi = cluster_bootstrap(dd, 3000); rows.append({"analysis": name, "endpoint": col, "n_tasks": len(dd), "estimate": est, "ci_low": lo, "ci_high": hi, "perm_p": permutation_p(dd, 10000)})
13 + add("primary (task-weighted, Y)", df)
14 + add("correct regardless of envelope", df, "Y_any")
15 + add("run-level unweighted mean difference", df.assign(task_id=df["run_id"])) # treats runs as independent — reported as an anti-pattern comparison
16 + for f in sorted(df["family"].unique()):
17 + add(f"leave-out family {f}", df[df["family"] != f])
18 + df2 = df.copy(); df2["Y"] = ((df2["scientific_success"] == 1) & (df2["n_violations"] == 0)).astype(int); add("ignoring envelope and status", df2)
19 + df3 = df.copy(); df3["Y"] = ((df3["q_correctness"] >= 0.999)).astype(int); add("all criteria passed (violations ignored)", df3)
20 + pd.DataFrame(rows).to_csv(os.path.join(EST, f"{split}_robustness.csv"), index=False)
21 + # order and node effects
22 + df["half"] = np.where(df["order_in_block"] < df.groupby("block_id")["order_in_block"].transform("max") / 2, "early", "late")
23 + pd.concat([df.groupby(["treatment", "half"])["Y"].mean().rename("Y").reset_index().assign(factor="order"), df.groupby(["treatment", "node"])["Y"].mean().rename("Y").reset_index().rename(columns={"node": "half"}).assign(factor="node")]).to_csv(os.path.join(EST, f"{split}_order_node.csv"), index=False)
24 + print(pd.DataFrame(rows).round(3).to_string())
25 +
26 +if __name__ == "__main__":
27 + main(sys.argv[1] if len(sys.argv) > 1 else "confirmatory")
added analysis/tables.py +61 −0
@@ -0,0 +1,61 @@
1 +"""LaTeX tables generated from the estimates (no manual numeric editing)."""
2 +import os, sys
3 +sys.path.insert(0, os.path.dirname(__file__))
4 +from common import *
5 +TAB = os.path.join(REPO, "results", "tables"); os.makedirs(TAB, exist_ok=True)
6 +
7 +def fmt(x, nd=3):
8 + if x is None or (isinstance(x, float) and np.isnan(x)): return "--"
9 + return f"{x:.{nd}f}" if isinstance(x, (float, np.floating)) else str(x)
10 +
11 +def table_arms(split):
12 + at = pd.read_csv(os.path.join(EST, f"{split}_arm_table.csv"))
13 + lines = [r"\begin{tabular}{lcccccccc}", r"\toprule", r"Arm & Runs & Tasks & $\hat p_a$ & Correct (any) & Viol.\ rate & Repro. & Quality & Cost/run (USD) \\", r"\midrule"]
14 + for _, r in at.iterrows():
15 + lines.append(f"{r['arm']} & {int(r['n_runs'])} & {int(r['n_tasks'])} & {fmt(r['Y_task_weighted'])} & {fmt(r['correct_any_task_weighted'])} & {fmt(r['violation_rate'])} & {fmt(r['reproducible_rate'])} & {fmt(r['quality_mean'])} & {fmt(r['cost_mean_usd'],2)} \\\\")
16 + lines += [r"\bottomrule", r"\end{tabular}"]
17 + open(os.path.join(TAB, f"tab_arms_{split}.tex"), "w").write("\n".join(lines) + "\n")
18 +
19 +def table_contrasts(split):
20 + ct = pd.read_csv(os.path.join(EST, f"{split}_contrasts.csv"))
21 + lines = [r"\begin{tabular}{llcccccc}", r"\toprule", r"Endpoint & Contrast & Role & $J$ & $\hat\Delta$ & 95\% CI & Perm.\ $p$ & Holm $p$ \\", r"\midrule"]
22 + names = {"Y": "Primary $Y$", "Y_any": "Correct (any)", "viol": "Violation", "quality_score": "Quality"}
23 + for _, r in ct.iterrows():
24 + lines.append(f"{names.get(r['endpoint'], r['endpoint'])} & {r['contrast']} & {r['role']} & {int(r['n_task_clusters'])} & {fmt(r['estimate'])} & [{fmt(r['ci_low'])}, {fmt(r['ci_high'])}] & {fmt(r['perm_p'])} & {fmt(r['holm_p']) if not pd.isna(r['holm_p']) else '--'} \\\\")
25 + lines += [r"\bottomrule", r"\end{tabular}"]
26 + open(os.path.join(TAB, f"tab_contrasts_{split}.tex"), "w").write("\n".join(lines) + "\n")
27 +
28 +def table_cost(split):
29 + at = pd.read_csv(os.path.join(EST, f"{split}_arm_table.csv"))
30 + lines = [r"\begin{tabular}{lccccccccc}", r"\toprule", r"Arm & Mean & Median & P90 & Total & Reviewer & Wall med.\ (s) & Wall P90 & Timeouts & Cost/success \\", r"\midrule"]
31 + for _, r in at.iterrows():
32 + lines.append(f"{r['arm']} & {fmt(r['cost_mean_usd'],2)} & {fmt(r['cost_median_usd'],2)} & {fmt(r['cost_p90_usd'],2)} & {fmt(r['cost_total_usd'],2)} & {fmt(r['reviewer_usd_total'],2)} & {fmt(r['wall_median_s'],0)} & {fmt(r['wall_p90_s'],0)} & {int(r['n_timeout'])} & {fmt(r['cost_per_success_usd'],2) if not pd.isna(r['cost_per_success_usd']) else 'undefined'} \\\\")
33 + lines += [r"\bottomrule", r"\end{tabular}"]
34 + open(os.path.join(TAB, f"tab_cost_{split}.tex"), "w").write("\n".join(lines) + "\n")
35 +
36 +def table_family(split):
37 + fam = pd.read_csv(os.path.join(EST, f"{split}_family_table.csv")); fams = sorted(fam["family"].unique())
38 + lines = [r"\begin{tabular}{l" + "c" * (2 * len(ARMS) + 1) + "}", r"\toprule", "Family & Tasks & " + " & ".join(f"$\\hat p_{a}$ & viol.\\ {a}" for a in ARMS) + r" \\", r"\midrule"]
39 + for f in fams:
40 + d = fam[fam["family"] == f].set_index("treatment")
41 + lines.append(f"{f} & {int(d['n_tasks'].iloc[0])} & " + " & ".join(f"{fmt(d.loc[a, 'Y'],2) if a in d.index else '--'} & {fmt(d.loc[a, 'viol'],2) if a in d.index else '--'}" for a in ARMS) + r" \\")
42 + lines += [r"\bottomrule", r"\end{tabular}"]
43 + open(os.path.join(TAB, f"tab_family_{split}.tex"), "w").write("\n".join(lines) + "\n")
44 +
45 +def table_accounting():
46 + df = pd.read_csv(os.path.join(DATA, "runs.csv"))
47 + lines = [r"\begin{tabular}{lcccccccc}", r"\toprule", r"Split & Assigned & Launched & Completed & Timeout & Budget & Infra.\ error & Evaluated & Successes \\", r"\midrule"]
48 + for s in ["development", "pilot", "confirmatory", "transfer", "recovery", "ablation"]:
49 + d = df[df["split"] == s]
50 + if d.empty: continue
51 + tr = d["terminal_reason"].fillna("")
52 + lines.append(f"{s} & {len(d)} & {int((d['status'] != 'not_started').sum())} & {int(tr.str.startswith('executor_finished').sum() + tr.eq('max_turns_reached').sum())} & {int(tr.eq('wall_clock_limit').sum())} & {int(tr.str.contains('budget').sum())} & {int(tr.str.startswith('infrastructure').sum())} & {int((d['status'] == 'evaluated').sum())} & {int((d['scientific_success'] == 1).sum())} \\\\")
53 + lines += [r"\bottomrule", r"\end{tabular}"]
54 + open(os.path.join(TAB, "tab_accounting.tex"), "w").write("\n".join(lines) + "\n")
55 +
56 +if __name__ == "__main__":
57 + split = sys.argv[1] if len(sys.argv) > 1 else "confirmatory"
58 + for f in (table_arms, table_contrasts, table_cost, table_family):
59 + try: f(split); print("ok", f.__name__)
60 + except Exception as e: print("skip", f.__name__, e)
61 + table_accounting()
added evaluators/run_validations.py +21 −0
@@ -0,0 +1,21 @@
1 +"""Run every oracle's validation/validate.py (seeded correct/wrong artefacts) and write reports/evaluator_validation.md."""
2 +import os, sys, subprocess, glob, json, datetime, re
3 +REPO = os.path.abspath(os.path.join(os.path.dirname(__file__), "..")); PY = os.path.join(REPO, ".venv", "bin", "python")
4 +rows = []
5 +for d in sorted(glob.glob(os.path.expanduser("~/wp29-oracles/F*"))):
6 + tid = os.path.basename(d); v = os.path.join(d, "validation", "validate.py")
7 + if not os.path.exists(v):
8 + rows.append((tid, "no validate.py", "", 0, 0)); continue
9 + r = subprocess.run([PY, v], capture_output=True, text=True, timeout=900, cwd=d)
10 + out = (r.stdout + r.stderr)
11 + if r.returncode != 0: # some validators take the repository root as argument
12 + r = subprocess.run([PY, v, REPO], capture_output=True, text=True, timeout=900, cwd=d); out = (r.stdout + r.stderr)
13 + n_cases = len(re.findall(r"(?im)^\s*(?:\[?(?:ok|pass|PASS|OK)\]?|✓|✔)", out)) or out.lower().count("ok")
14 + status = "PASS" if r.returncode == 0 and re.search(r"ALL (OK|PASS)|VALIDATION PASSED|all cases ok|passed", out, re.I) else ("PASS(exit0)" if r.returncode == 0 else "FAIL")
15 + rows.append((tid, status, out.strip().splitlines()[-1][:160] if out.strip() else "", r.returncode, len(out.splitlines())))
16 + print(tid, status)
17 +lines = ["# Evaluator validation (seeded artefacts)", "", f"Generated {datetime.datetime.now(datetime.timezone.utc).isoformat(timespec='seconds')} by `evaluators/run_validations.py`. Each hidden oracle ships a `validation/` set with ≥1 correct_out (an independently written solution whose analysis.py is re-run) and ≥2 wrong_out artefacts (planted scientific errors, fabrication, false completion or leakage). The evaluator must accept the former and reject the latter; a PASS means every seeded case received the expected verdict (false acceptance = 0 and false rejection = 0 on the seeded set). The seeded set is small and designer-authored; it bounds gross evaluator errors, not the full space of valid solutions.", "", "| task_id | result | last line of validation output |", "|---|---|---|"]
18 +for tid, st, last, rc, n in rows: lines.append(f"| {tid} | {st} | {last.replace('|', '/')} |")
19 +lines += ["", f"Summary: {sum(1 for r in rows if r[1].startswith('PASS'))} PASS, {sum(1 for r in rows if r[1]=='FAIL')} FAIL, {sum(1 for r in rows if 'no validate' in r[1])} without validation set, of {len(rows)} oracles."]
20 +open(os.path.join(REPO, "reports", "evaluator_validation.md"), "w").write("\n".join(lines) + "\n")
21 +print(lines[-1])
added paper/appendices/appendix.tex +1 −0
@@ -0,0 +1 @@
1 +% appendices
added paper/figures/fig_architecture.pdf +0 −0

Binary file not shown.

added paper/main.tex +149 −0
@@ -0,0 +1,149 @@
1 +% ============================================================================
2 +% UQO Working Paper Template — Economics / Finance
3 +% ============================================================================
4 +\documentclass[12pt,letterpaper]{article}
5 +
6 +% --- Encoding & Language ---
7 +\usepackage[utf8]{inputenc}
8 +\usepackage[T1]{fontenc}
9 +\usepackage[english]{babel}
10 +
11 +% --- Page Layout ---
12 +\usepackage[
13 + letterpaper,
14 + top=1in,
15 + bottom=1in,
16 + left=1in,
17 + right=1in
18 +]{geometry}
19 +
20 +% --- Typography ---
21 +\usepackage{mathptmx} % Times-like font (text + math)
22 +\usepackage{microtype} % Improved typographic refinements
23 +\usepackage{setspace}
24 +\onehalfspacing % 1.5 line spacing
25 +
26 +% --- Math ---
27 +\usepackage{amsmath,amssymb,amsthm}
28 +
29 +% --- Tables ---
30 +\usepackage{booktabs}
31 +\usepackage{tabularx}
32 +\usepackage{multirow}
33 +\usepackage{threeparttable}
34 +\usepackage{siunitx}
35 +\sisetup{
36 + input-symbols = {()},
37 + table-format = -1.3,
38 + parse-numbers = false
39 +}
40 +
41 +% --- Figures ---
42 +\usepackage{graphicx}
43 +\usepackage{subcaption}\n\usepackage{algorithm}\n\usepackage{algpseudocode}\n\usepackage{longtable}\n\usepackage{array}\n\usepackage{adjustbox}
44 +\usepackage{float}
45 +
46 +% --- Colors & Links ---
47 +\usepackage[dvipsnames]{xcolor}
48 +\usepackage[
49 + colorlinks = true,
50 + linkcolor = NavyBlue,
51 + citecolor = NavyBlue,
52 + urlcolor = NavyBlue,
53 + breaklinks = true
54 +]{hyperref}
55 +
56 +% --- Bibliography ---
57 +\usepackage[authoryear,round,semicolon]{natbib}
58 +\bibliographystyle{aer}
59 +
60 +% --- Captions ---
61 +\usepackage[
62 + font = small,
63 + labelfont = bf,
64 + labelsep = period,
65 + justification = justified,
66 + singlelinecheck = false
67 +]{caption}
68 +
69 +% --- Headers & Footers ---
70 +\usepackage{fancyhdr}
71 +\pagestyle{fancy}
72 +\fancyhf{}
73 +\renewcommand{\headrulewidth}{0pt}
74 +\fancyfoot[C]{\thepage}
75 +
76 +% --- Section Formatting ---
77 +\usepackage{titlesec}
78 +\titleformat{\section}{\large\bfseries}{\thesection.}{0.5em}{}
79 +\titleformat{\subsection}{\normalsize\bfseries}{\thesubsection.}{0.5em}{}
80 +\titleformat{\subsubsection}{\normalsize\itshape}{\thesubsubsection.}{0.5em}{}
81 +
82 +% --- Appendix Support ---
83 +\usepackage[toc,page]{appendix}
84 +
85 +% --- Misc ---
86 +\usepackage{enumitem}
87 +\setlist{nosep,leftmargin=*}
88 +
89 +% ============================================================================
90 +% METADATA — Edit these fields for each working paper
91 +% ============================================================================
92 +\newcommand{\WPnumber}{29}
93 +\newcommand{\WPtitle}{Protocol-Guided Autonomous Research with Claude Code}
94 +\newcommand{\WPsubtitle}{Scientific Verification, Reproducibility, and the Cost of Delegation: A Controlled Study}
95 +\newcommand{\WPdate}{October 2026}
96 +\newcommand{\WPversion}{1.0}
97 +\newcommand{\WPabstract}{%
98 + This paper examines \ldots{}
99 + We find that \ldots{}
100 + Our results suggest \ldots{}
101 + These findings contribute to the literature on \ldots{}
102 +}
103 +\newcommand{\WPkeywords}{autonomous research agents; Claude Code; research protocols; executable verification; reproducibility; agent evaluation; delegation}
104 +\newcommand{\WPjel}{C18, C63, C88, O33}
105 +
106 +% --- Author ---
107 +\newcommand{\WPauthor}{Simon-Pierre Boucher}
108 +\newcommand{\WPaffiliation}{%
109 + Département des sciences administratives\\
110 + Université du Québec en Outaouais%
111 +}
112 +\newcommand{\WPemail}{simon-pierre.boucher@uqo.ca}
113 +\newcommand{\WPaddress}{%
114 + Gatineau -- Pavillon Alexandre-Taché\\
115 + 283, boulevard Alexandre-Taché\\
116 + Gatineau, Québec, Canada J9A 1L8%
117 +}
118 +
119 +% ============================================================================
120 +% DOCUMENT
121 +% ============================================================================
122 +\begin{document}
123 +
124 +% --- Title Page ---
125 +\input{sections/titlepage}
126 +
127 +% --- Main Body ---
128 +\input{sections/introduction}
129 +\input{sections/literature}
130 +\input{sections/framework}
131 +\input{sections/corpus}
132 +\input{sections/implementation}
133 +\input{sections/design}
134 +\input{sections/evaluation}
135 +\input{sections/results}
136 +\input{sections/recovery}
137 +\input{sections/transfer}
138 +\input{sections/reflexive}
139 +\input{sections/limitations}
140 +\input{sections/conclusion}
141 +
142 +% --- References ---
143 +\newpage
144 +\bibliography{references}
145 +
146 +% --- Appendix ---
147 +\input{appendices/appendix}
148 +
149 +\end{document}
added paper/references.bib +915 −0
@@ -0,0 +1,915 @@
1 +@misc{lu2024aiscientist,
2 + author = {Chris Lu and Cong Lu and Robert Tjarko Lange and Jakob Foerster and Jeff Clune and David Ha},
3 + title = {{The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery}},
4 + year = {2024},
5 + howpublished = {arXiv preprint arXiv:2408.06292},
6 + eprint = {2408.06292},
7 + archivePrefix = {arXiv}
8 +}
9 +% verified via https://export.arxiv.org/api/query?id_list=2408.06292 on 2026-10-30 (comment placed after the first entry so that no % line precedes it)
10 +
11 +% verified via https://api.crossref.org/works/10.1038/s41586-026-10265-5 on 2026-10-30
12 +@article{lu2026aiscientist,
13 + author = {Lu, Chris and Lu, Cong and Lange, Robert Tjarko and Yamada, Yutaro and Hu, Shengran and Foerster, Jakob and Ha, David and Clune, Jeff},
14 + title = {{Towards end-to-end automation of AI research}},
15 + year = {2026},
16 + journal = {Nature},
17 + volume = {651},
18 + number = {8107},
19 + pages = {914--919},
20 + doi = {10.1038/s41586-026-10265-5}
21 +}
22 +
23 +% verified via https://export.arxiv.org/api/query?id_list=2504.08066 on 2026-10-30
24 +@misc{yamada2025aiscientistv2,
25 + author = {Yutaro Yamada and Robert Tjarko Lange and Cong Lu and Shengran Hu and Chris Lu and Jakob Foerster and Jeff Clune and David Ha},
26 + title = {{The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search}},
27 + year = {2025},
28 + howpublished = {arXiv preprint arXiv:2504.08066},
29 + eprint = {2504.08066},
30 + archivePrefix = {arXiv}
31 +}
32 +
33 +% verified via https://api.crossref.org/works/10.18653/v1/2025.findings-emnlp.320 on 2026-10-30
34 +@inproceedings{schmidgall2025agentlab,
35 + author = {Schmidgall, Samuel and Su, Yusheng and Wang, Ze and Sun, Ximeng and Wu, Jialian and Yu, Xiaodong and Liu, Jiang and Moor, Michael and Liu, Zicheng and Barsoum, Emad},
36 + title = {{Agent Laboratory: Using LLM Agents as Research Assistants}},
37 + year = {2025},
38 + booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2025},
39 + pages = {5977--6043},
40 + doi = {10.18653/v1/2025.findings-emnlp.320},
41 + eprint = {2501.04227},
42 + archivePrefix = {arXiv}
43 +}
44 +
45 +% verified via https://api.crossref.org/works/10.1038/s41586-023-06792-0 on 2026-10-30
46 +@article{boiko2023coscientist,
47 + author = {Boiko, Daniil A. and MacKnight, Robert and Kline, Ben and Gomes, Gabe},
48 + title = {{Autonomous chemical research with large language models}},
49 + year = {2023},
50 + journal = {Nature},
51 + volume = {624},
52 + number = {7992},
53 + pages = {570--578},
54 + doi = {10.1038/s41586-023-06792-0}
55 +}
56 +
57 +% verified via https://api.crossref.org/works/10.1038/s42256-024-00832-8 on 2026-10-30
58 +@article{bran2024chemcrow,
59 + author = {M. Bran, Andres and Cox, Sam and Schilter, Oliver and Baldassari, Carlo and White, Andrew D. and Schwaller, Philippe},
60 + title = {{Augmenting large language models with chemistry tools}},
61 + year = {2024},
62 + journal = {Nature Machine Intelligence},
63 + volume = {6},
64 + number = {5},
65 + pages = {525--535},
66 + doi = {10.1038/s42256-024-00832-8}
67 +}
68 +
69 +% verified via https://api.crossref.org/works/10.1056/AIoa2400555 on 2026-10-30
70 +@article{ifargan2025datatopaper,
71 + author = {Ifargan, Tal and Hafner, Lukas and Kern, Maor and Alcalay, Ori and Kishony, Roy},
72 + title = {{Autonomous LLM-Driven Research — from Data to Human-Verifiable Research Papers}},
73 + year = {2025},
74 + journal = {NEJM AI},
75 + volume = {2},
76 + number = {1},
77 + doi = {10.1056/AIoa2400555},
78 + eprint = {2404.17605},
79 + archivePrefix = {arXiv}
80 +}
81 +
82 +% verified via https://api.crossref.org/works/10.1038/s41586-026-10644-y on 2026-10-30
83 +@article{gottweis2026coscientist,
84 + author = {Gottweis, Juraj and Weng, Wei-Hung and Daryin, Alexander and Tu, Tao and Sirkovic, Petar and Myaskovsky, Artiom and Glowaty, Grzegorz and Weissenberger, Felix and Orlandi, Alessio and Popovici, Dan and Palepu, Anil and Rong, Keran and Tanno, Ryutaro and Saab, Khaled and Zhang, Fan and Blum, Jacob and Carroll, Andrew and Kulkarni, Kavita and Tomašev, Nenad and Zverinski, Dina and Rendulic, Ivor and Vedadi, Elahe and Hasler, Florian and Rimanic, Luka and Boia, Marina and Budiselic, Ivan and Feinstein, Ben and Bellaiche, Mathias and Sheffer, Tom and Freyberg, Jan and Ratcliff, Jeremy and Bertolli, Ottavia and Chou, Katherine and Hassidim, Avinatan and Gokturk, Burak and Vahdat, Amin and Guan, Yuan and Dhillon, Vikram and Vaishnav, Eeshit Dhaval and Lee, Byron and Costa, Tiago R. D. and Penadés, José R. and Peltz, Gary and Matias, Yossi and Manyika, James and Hassabis, Demis and Xu, Yunhan and Kohli, Pushmeet and Pawlosky, Annalisa and Karthikesalingam, Alan and Natarajan, Vivek},
85 + title = {{Accelerating scientific discovery with Co-Scientist}},
86 + year = {2026},
87 + journal = {Nature},
88 + volume = {655},
89 + number = {8122},
90 + pages = {487--496},
91 + doi = {10.1038/s41586-026-10644-y},
92 + eprint = {2502.18864},
93 + archivePrefix = {arXiv}
94 +}
95 +
96 +% verified via https://api.crossref.org/works/10.1038/s41586-025-09442-9 on 2026-10-30
97 +@article{swanson2025virtuallab,
98 + author = {Swanson, Kyle and Wu, Wesley and Bulaong, Nash L. and Pak, John E. and Zou, James},
99 + title = {{The Virtual Lab of AI agents designs new SARS-CoV-2 nanobodies}},
100 + year = {2025},
101 + journal = {Nature},
102 + volume = {646},
103 + number = {8085},
104 + pages = {716--723},
105 + doi = {10.1038/s41586-025-09442-9}
106 +}
107 +
108 +% verified via https://export.arxiv.org/api/query?id_list=2511.02824 on 2026-10-30
109 +@misc{mitchener2025kosmos,
110 + author = {Ludovico Mitchener and Angela Yiu and Benjamin Chang and Mathieu Bourdenx and Tyler Nadolski and Arvis Sulovari and Eric C. Landsness and Daniel L. Barabasi and Siddharth Narayanan and Nicky Evans and Shriya Reddy and Martha Foiani and Aizad Kamal and Leah P. Shriver and Fang Cao and Asmamaw T. Wassie and Jon M. Laurent and Edwin Melville-Green and Mayk Caldas and Albert Bou and Kaleigh F. Roberts and Sladjana Zagorac and Timothy C. Orr and Miranda E. Orr and Kevin J. Zwezdaryk and Ali E. Ghareeb and Laurie McCoy and Bruna Gomes and Euan A. Ashley and Karen E. Duff and Tonio Buonassisi and Tom Rainforth and Randall J. Bateman and Michael Skarlinski and Samuel G. Rodriques and Michaela M. Hinks and Andrew D. White},
111 + title = {{Kosmos: An AI Scientist for Autonomous Discovery}},
112 + year = {2025},
113 + howpublished = {arXiv preprint arXiv:2511.02824},
114 + eprint = {2511.02824},
115 + archivePrefix = {arXiv}
116 +}
117 +
118 +% verified via https://export.arxiv.org/api/query?id_list=2509.08713 on 2026-10-30
119 +@misc{luo2025pitfalls,
120 + author = {Ziming Luo and Atoosa Kasirzadeh and Nihar B. Shah},
121 + title = {{The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systems}},
122 + year = {2025},
123 + howpublished = {arXiv preprint arXiv:2509.08713},
124 + eprint = {2509.08713},
125 + archivePrefix = {arXiv}
126 +}
127 +
128 +% verified via https://api.crossref.org/works/10.1145/3769733.3769747 on 2026-10-30
129 +@article{beel2025sakana,
130 + author = {Beel, Joeran and Kan, Min-Yen and Baumgart, Moritz},
131 + title = {{Evaluating Sakana's AI Scientist: Bold Claims, Mixed Results, and a Promising Future?}},
132 + year = {2025},
133 + journal = {ACM SIGIR Forum},
134 + volume = {59},
135 + number = {1},
136 + pages = {1--20},
137 + doi = {10.1145/3769733.3769747},
138 + eprint = {2502.14297},
139 + archivePrefix = {arXiv}
140 +}
141 +
142 +% verified via https://export.arxiv.org/api/query?id_list=2607.12301 on 2026-10-30
143 +@misc{luo2026xscientist,
144 + author = {Jixiang Luo},
145 + title = {{XScientist: A Git-Like Research Protocol for Long-Running Autonomous Scientific Discovery}},
146 + year = {2026},
147 + howpublished = {arXiv preprint arXiv:2607.12301},
148 + eprint = {2607.12301},
149 + archivePrefix = {arXiv}
150 +}
151 +
152 +% verified via https://export.arxiv.org/api/query?id_list=2608.05179 on 2026-10-30
153 +@misc{ding2026verificationgap,
154 + author = {Tianyu Ding and Aditya Nannapaneni and Bingfan Liu and Ling Zhang},
155 + title = {{Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap}},
156 + year = {2026},
157 + howpublished = {arXiv preprint arXiv:2608.05179},
158 + eprint = {2608.05179},
159 + archivePrefix = {arXiv}
160 +}
161 +
162 +% verified via https://export.arxiv.org/api/query?id_list=2410.05080 on 2026-10-30
163 +@inproceedings{chen2025scienceagentbench,
164 + author = {Ziru Chen and Shijie Chen and Yuting Ning and Qianheng Zhang and Boshi Wang and Botao Yu and Yifei Li and Zeyi Liao and Chen Wei and Zitong Lu and Vishal Dey and Mingyi Xue and Frazier N. Baker and Benjamin Burns and Daniel Adu-Ampratwum and Xuhui Huang and Xia Ning and Song Gao and Yu Su and Huan Sun},
165 + title = {{ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery}},
166 + year = {2024},
167 + booktitle = {ICLR 2025},
168 + eprint = {2410.05080},
169 + archivePrefix = {arXiv}
170 +}
171 +
172 +% verified via https://export.arxiv.org/api/query?id_list=2407.01725 on 2026-10-30
173 +@inproceedings{majumder2025discoverybench,
174 + author = {Bodhisattwa Prasad Majumder and Harshit Surana and Dhruv Agarwal and Bhavana Dalvi Mishra and Abhijeetsingh Meena and Aryan Prakhar and Tirth Vora and Tushar Khot and Ashish Sabharwal and Peter Clark},
175 + title = {{DiscoveryBench: Towards Data-Driven Discovery with Large Language Models}},
176 + year = {2024},
177 + booktitle = {ICLR 2025},
178 + eprint = {2407.01725},
179 + archivePrefix = {arXiv}
180 +}
181 +
182 +% verified via https://export.arxiv.org/api/query?id_list=2409.07703 on 2026-10-30
183 +@inproceedings{jing2025dsbench,
184 + author = {Liqiang Jing and Zhehui Huang and Xiaoyang Wang and Wenlin Yao and Wenhao Yu and Kaixin Ma and Hongming Zhang and Xinya Du and Dong Yu},
185 + title = {{DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?}},
186 + year = {2024},
187 + booktitle = {ICLR 2025},
188 + eprint = {2409.07703},
189 + archivePrefix = {arXiv}
190 +}
191 +
192 +% verified via https://export.arxiv.org/api/query?id_list=2310.03302 on 2026-10-30
193 +@inproceedings{huang2024mlagentbench,
194 + author = {Qian Huang and Jian Vora and Percy Liang and Jure Leskovec},
195 + title = {{MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation}},
196 + year = {2023},
197 + booktitle = {ICML 2024},
198 + eprint = {2310.03302},
199 + archivePrefix = {arXiv}
200 +}
201 +
202 +% verified via https://export.arxiv.org/api/query?id_list=2408.09667 on 2026-10-30
203 +@inproceedings{gu2024blade,
204 + author = {Ken Gu and Ruoxi Shang and Ruien Jiang and Keying Kuang and Richard-John Lin and Donghe Lyu and Yue Mao and Youran Pan and Teng Wu and Jiaqian Yu and Yikun Zhang and Tianmai M. Zhang and Lanyi Zhu and Mike A. Merrill and Jeffrey Heer and Tim Althoff},
205 + title = {{BLADE: Benchmarking Language Model Agents for Data-Driven Science}},
206 + year = {2024},
207 + booktitle = {EMNLP 2024},
208 + eprint = {2408.09667},
209 + archivePrefix = {arXiv}
210 +}
211 +
212 +% verified via https://export.arxiv.org/api/query?id_list=2310.06770 on 2026-10-30
213 +@inproceedings{jimenez2024swebench,
214 + author = {Carlos E. Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik Narasimhan},
215 + title = {{SWE-bench: Can Language Models Resolve Real-World GitHub Issues?}},
216 + year = {2023},
217 + booktitle = {ICLR 2024},
218 + eprint = {2310.06770},
219 + archivePrefix = {arXiv}
220 +}
221 +
222 +% verified via https://export.arxiv.org/api/query?id_list=2410.07095 on 2026-10-30
223 +@inproceedings{chan2025mlebench,
224 + author = {Jun Shern Chan and Neil Chowdhury and Oliver Jaffe and James Aung and Dane Sherburn and Evan Mays and Giulio Starace and Kevin Liu and Leon Maksin and Tejal Patwardhan and Lilian Weng and Aleksander Mądry},
225 + title = {{MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering}},
226 + year = {2024},
227 + booktitle = {ICLR 2025},
228 + eprint = {2410.07095},
229 + archivePrefix = {arXiv}
230 +}
231 +
232 +% verified via https://export.arxiv.org/api/query?id_list=2409.11363 on 2026-10-30
233 +@article{siegel2024corebench,
234 + author = {Zachary S. Siegel and Sayash Kapoor and Nitya Nadgir and Benedikt Stroebl and Arvind Narayanan},
235 + title = {{CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark}},
236 + year = {2024},
237 + journal = {Transactions on Machine Learning Research},
238 + eprint = {2409.11363},
239 + archivePrefix = {arXiv}
240 +}
241 +
242 +% verified via https://export.arxiv.org/api/query?id_list=2504.01848 on 2026-10-30
243 +@inproceedings{starace2025paperbench,
244 + author = {Giulio Starace and Oliver Jaffe and Dane Sherburn and James Aung and Jun Shern Chan and Leon Maksin and Rachel Dias and Evan Mays and Benjamin Kinsella and Wyatt Thompson and Johannes Heidecke and Amelia Glaese and Tejal Patwardhan},
245 + title = {{PaperBench: Evaluating AI's Ability to Replicate AI Research}},
246 + year = {2025},
247 + booktitle = {ICML 2025},
248 + eprint = {2504.01848},
249 + archivePrefix = {arXiv}
250 +}
251 +
252 +% verified via https://api.crossref.org/works/10.1093/bioinformatics/bts480 on 2026-10-30
253 +@article{koster2012snakemake,
254 + author = {Köster, Johannes and Rahmann, Sven},
255 + title = {{Snakemake—a scalable bioinformatics workflow engine}},
256 + year = {2012},
257 + journal = {Bioinformatics},
258 + volume = {28},
259 + number = {19},
260 + pages = {2520--2522},
261 + doi = {10.1093/bioinformatics/bts480}
262 +}
263 +
264 +% verified via https://api.crossref.org/works/10.1038/nbt.3820 on 2026-10-30
265 +@article{ditommaso2017nextflow,
266 + author = {Di Tommaso, Paolo and Chatzou, Maria and Floden, Evan W and Barja, Pablo Prieto and Palumbo, Emilio and Notredame, Cedric},
267 + title = {{Nextflow enables reproducible computational workflows}},
268 + year = {2017},
269 + journal = {Nature Biotechnology},
270 + volume = {35},
271 + number = {4},
272 + pages = {316--319},
273 + doi = {10.1038/nbt.3820}
274 +}
275 +
276 +% verified via https://api.crossref.org/works/10.1145/2452376.2452478 on 2026-10-30
277 +@inproceedings{missier2013prov,
278 + author = {Missier, Paolo and Belhajjame, Khalid and Cheney, James},
279 + title = {{The W3C PROV family of specifications for modelling provenance metadata}},
280 + year = {2013},
281 + booktitle = {Proceedings of the 16th International Conference on Extending Database Technology},
282 + pages = {773--776},
283 + doi = {10.1145/2452376.2452478}
284 +}
285 +
286 +% verified via https://api.crossref.org/works/10.1145/2882903.2899401 on 2026-10-30
287 +@inproceedings{chirigati2016reprozip,
288 + author = {Chirigati, Fernando and Rampin, Rémi and Shasha, Dennis and Freire, Juliana},
289 + title = {{ReproZip}},
290 + year = {2016},
291 + booktitle = {Proceedings of the 2016 International Conference on Management of Data},
292 + pages = {2085--2088},
293 + doi = {10.1145/2882903.2899401}
294 +}
295 +
296 +% verified via https://api.crossref.org/works/10.1371/journal.pcbi.1003285 on 2026-10-30
297 +@article{sandve2013tensimplerules,
298 + author = {Sandve, Geir Kjetil and Nekrutenko, Anton and Taylor, James and Hovig, Eivind},
299 + title = {{Ten Simple Rules for Reproducible Computational Research}},
300 + year = {2013},
301 + journal = {PLoS Computational Biology},
302 + volume = {9},
303 + number = {10},
304 + pages = {e1003285},
305 + doi = {10.1371/journal.pcbi.1003285}
306 +}
307 +
308 +% verified via https://api.crossref.org/works/10.1073/pnas.1708290115 on 2026-10-30
309 +@article{stodden2018journalpolicy,
310 + author = {Stodden, Victoria and Seiler, Jennifer and Ma, Zhaokun},
311 + title = {{An empirical analysis of journal policy effectiveness for computational reproducibility}},
312 + year = {2018},
313 + journal = {Proceedings of the National Academy of Sciences},
314 + volume = {115},
315 + number = {11},
316 + pages = {2584--2589},
317 + doi = {10.1073/pnas.1708290115}
318 +}
319 +
320 +% verified via https://api.crossref.org/works/10.1126/science.1213847 on 2026-10-30
321 +@article{peng2011reproducible,
322 + author = {Peng, Roger D.},
323 + title = {{Reproducible Research in Computational Science}},
324 + year = {2011},
325 + journal = {Science},
326 + volume = {334},
327 + number = {6060},
328 + pages = {1226--1227},
329 + doi = {10.1126/science.1213847}
330 +}
331 +
332 +% verified via https://api.crossref.org/works/10.17226/25303 on 2026-10-30
333 +@book{nasem2019reproducibility,
334 + author = {{National Academies of Sciences, Engineering, and Medicine}},
335 + title = {{Reproducibility and Replicability in Science}},
336 + year = {2019},
337 + publisher = {The National Academies Press},
338 + address = {Washington, DC},
339 + doi = {10.17226/25303}
340 +}
341 +
342 +% verified via https://api.crossref.org/works/10.1098/rsta.2020.0069 on 2026-10-30
343 +@article{krafczyk2021reproduction,
344 + author = {Krafczyk, M. S. and Shi, A. and Bhaskar, A. and Marinov, D. and Stodden, V.},
345 + title = {{Learning from reproducing computational results: introducing three principles and the <i>Reproduction Package</i>}},
346 + year = {2021},
347 + journal = {Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences},
348 + volume = {379},
349 + number = {2197},
350 + doi = {10.1098/rsta.2020.0069}
351 +}
352 +
353 +% verified via https://api.crossref.org/works/10.1038/s41597-022-01143-6 on 2026-10-30
354 +@article{trisovic2022codequality,
355 + author = {Trisovic, Ana and Lau, Matthew K. and Pasquier, Thomas and Crosas, Mercè},
356 + title = {{A large-scale study on research code quality and execution}},
357 + year = {2022},
358 + journal = {Scientific Data},
359 + volume = {9},
360 + number = {1},
361 + doi = {10.1038/s41597-022-01143-6}
362 +}
363 +
364 +% verified via https://api.crossref.org/works/10.1073/pnas.1708274114 on 2026-10-30
365 +@article{nosek2018preregistration,
366 + author = {Nosek, Brian A. and Ebersole, Charles R. and DeHaven, Alexander C. and Mellor, David T.},
367 + title = {{The preregistration revolution}},
368 + year = {2018},
369 + journal = {Proceedings of the National Academy of Sciences},
370 + volume = {115},
371 + number = {11},
372 + pages = {2600--2606},
373 + doi = {10.1073/pnas.1708274114}
374 +}
375 +
376 +% verified via https://api.crossref.org/works/10.1016/j.cortex.2012.12.016 on 2026-10-30
377 +@article{chambers2013registeredreports,
378 + author = {Chambers, Christopher D.},
379 + title = {{Registered Reports: A new publishing initiative at Cortex}},
380 + year = {2013},
381 + journal = {Cortex},
382 + volume = {49},
383 + number = {3},
384 + pages = {609--610},
385 + doi = {10.1016/j.cortex.2012.12.016}
386 +}
387 +
388 +% verified via https://api.crossref.org/works/10.1038/s41562-021-01193-7 on 2026-10-30
389 +@article{chambers2022registeredreports,
390 + author = {Chambers, Christopher D. and Tzavella, Loukia},
391 + title = {{The past, present and future of Registered Reports}},
392 + year = {2021},
393 + journal = {Nature Human Behaviour},
394 + volume = {6},
395 + number = {1},
396 + pages = {29--42},
397 + doi = {10.1038/s41562-021-01193-7}
398 +}
399 +
400 +% verified via https://api.crossref.org/works/10.1177/0956797611417632 on 2026-10-30
401 +@article{simmons2011falsepositive,
402 + author = {Simmons, Joseph P. and Nelson, Leif D. and Simonsohn, Uri},
403 + title = {{False-Positive Psychology}},
404 + year = {2011},
405 + journal = {Psychological Science},
406 + volume = {22},
407 + number = {11},
408 + pages = {1359--1366},
409 + doi = {10.1177/0956797611417632}
410 +}
411 +
412 +% verified via https://api.crossref.org/works/10.1038/s41562-016-0021 on 2026-10-30
413 +@article{munafo2017manifesto,
414 + author = {Munafò, Marcus R. and Nosek, Brian A. and Bishop, Dorothy V. M. and Button, Katherine S. and Chambers, Christopher D. and Percie du Sert, Nathalie and Simonsohn, Uri and Wagenmakers, Eric-Jan and Ware, Jennifer J. and Ioannidis, John P. A.},
415 + title = {{A manifesto for reproducible science}},
416 + year = {2017},
417 + journal = {Nature Human Behaviour},
418 + volume = {1},
419 + number = {1},
420 + doi = {10.1038/s41562-016-0021}
421 +}
422 +
423 +% verified via https://api.crossref.org/works/10.1511/2014.111.460 on 2026-10-30
424 +@article{gelman2014statisticalcrisis,
425 + author = {Gelman, Andrew and Loken, Eric},
426 + title = {{The Statistical Crisis in Science}},
427 + year = {2014},
428 + journal = {American Scientist},
429 + volume = {102},
430 + number = {6},
431 + pages = {460},
432 + doi = {10.1511/2014.111.460}
433 +}
434 +
435 +% verified via https://api.crossref.org/works/10.1126/science.aab2374 on 2026-10-30
436 +@article{nosek2015top,
437 + author = {Nosek, B. A. and Alter, G. and Banks, G. C. and Borsboom, D. and Bowman, S. D. and Breckler, S. J. and Buck, S. and Chambers, C. D. and Chin, G. and Christensen, G. and Contestabile, M. and Dafoe, A. and Eich, E. and Freese, J. and Glennerster, R. and Goroff, D. and Green, D. P. and Hesse, B. and Humphreys, M. and Ishiyama, J. and Karlan, D. and Kraut, A. and Lupia, A. and Mabry, P. and Madon, T. and Malhotra, N. and Mayo-Wilson, E. and McNutt, M. and Miguel, E. and Paluck, E. Levy and Simonsohn, U. and Soderberg, C. and Spellman, B. A. and Turitto, J. and VandenBos, G. and Vazire, S. and Wagenmakers, E. J. and Wilson, R. and Yarkoni, T.},
438 + title = {{Promoting an open research culture}},
439 + year = {2015},
440 + journal = {Science},
441 + volume = {348},
442 + number = {6242},
443 + pages = {1422--1425},
444 + doi = {10.1126/science.aab2374}
445 +}
446 +
447 +% verified via https://api.crossref.org/works/10.1177/25152459211007467 on 2026-10-30
448 +@article{scheel2021excess,
449 + author = {Scheel, Anne M. and Schijen, Mitchell R. M. J. and Lakens, Daniël},
450 + title = {{An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports}},
451 + year = {2021},
452 + journal = {Advances in Methods and Practices in Psychological Science},
453 + volume = {4},
454 + number = {2},
455 + doi = {10.1177/25152459211007467}
456 +}
457 +
458 +% verified via https://api.crossref.org/works/10.1126/science.aac4716 on 2026-10-30
459 +@article{osc2015reproducibility,
460 + author = {{Open Science Collaboration}},
461 + title = {{Estimating the reproducibility of psychological science}},
462 + year = {2015},
463 + journal = {Science},
464 + volume = {349},
465 + number = {6251},
466 + doi = {10.1126/science.aac4716}
467 +}
468 +
469 +% verified via https://api.crossref.org/works/10.1207/s15327957pspr0203_4 on 2026-10-30
470 +@article{kerr1998harking,
471 + author = {Kerr, Norbert L.},
472 + title = {{HARKing: Hypothesizing After the Results are Known}},
473 + year = {1998},
474 + journal = {Personality and Social Psychology Review},
475 + volume = {2},
476 + number = {3},
477 + pages = {196--217},
478 + doi = {10.1207/s15327957pspr0203_4}
479 +}
480 +
481 +% verified via https://export.arxiv.org/api/query?id_list=2308.03688 on 2026-10-30
482 +@inproceedings{liu2024agentbench,
483 + author = {Xiao Liu and Hao Yu and Hanchen Zhang and Yifan Xu and Xuanyu Lei and Hanyu Lai and Yu Gu and Hangliang Ding and Kaiwen Men and Kejuan Yang and Shudan Zhang and Xiang Deng and Aohan Zeng and Zhengxiao Du and Chenhui Zhang and Sheng Shen and Tianjun Zhang and Yu Su and Huan Sun and Minlie Huang and Yuxiao Dong and Jie Tang},
484 + title = {{AgentBench: Evaluating LLMs as Agents}},
485 + year = {2023},
486 + booktitle = {ICLR 2024},
487 + eprint = {2308.03688},
488 + archivePrefix = {arXiv}
489 +}
490 +
491 +% verified via https://export.arxiv.org/api/query?id_list=2406.12045 on 2026-10-30
492 +@misc{yao2024taubench,
493 + author = {Shunyu Yao and Noah Shinn and Pedram Razavi and Karthik Narasimhan},
494 + title = {{$$\tau$$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains}},
495 + year = {2024},
496 + howpublished = {arXiv preprint arXiv:2406.12045},
497 + eprint = {2406.12045},
498 + archivePrefix = {arXiv}
499 +}
500 +
501 +% verified via https://export.arxiv.org/api/query?id_list=2407.01502 on 2026-10-30
502 +@article{kapoor2024agentsthatmatter,
503 + author = {Sayash Kapoor and Benedikt Stroebl and Zachary S. Siegel and Nitya Nadgir and Arvind Narayanan},
504 + title = {{AI Agents That Matter}},
505 + year = {2024},
506 + journal = {Transactions on Machine Learning Research},
507 + eprint = {2407.01502},
508 + archivePrefix = {arXiv}
509 +}
510 +
511 +% verified via https://api.crossref.org/works/10.18653/v1/2023.findings-emnlp.722 on 2026-10-30
512 +@inproceedings{sainz2023contamination,
513 + author = {Sainz, Oscar and Campos, Jon and García-Ferrero, Iker and Etxaniz, Julen and de Lacalle, Oier Lopez and Agirre, Eneko},
514 + title = {{NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark}},
515 + year = {2023},
516 + booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
517 + pages = {10776--10787},
518 + doi = {10.18653/v1/2023.findings-emnlp.722},
519 + eprint = {2310.18018},
520 + archivePrefix = {arXiv}
521 +}
522 +
523 +% verified via https://export.arxiv.org/api/query?id_list=2306.05685 on 2026-10-30
524 +@inproceedings{zheng2023mtbench,
525 + author = {Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric P. Xing and Hao Zhang and Joseph E. Gonzalez and Ion Stoica},
526 + title = {{Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena}},
527 + year = {2023},
528 + booktitle = {NeurIPS 2023 Datasets and Benchmarks},
529 + eprint = {2306.05685},
530 + archivePrefix = {arXiv}
531 +}
532 +
533 +% verified via https://export.arxiv.org/api/query?id_list=2406.12624 on 2026-10-30
534 +@inproceedings{thakur2025judgingjudges,
535 + author = {Aman Singh Thakur and Kartik Choudhary and Venkat Srinik Ramayapally and Sankaran Vaidyanathan and Dieuwke Hupkes},
536 + title = {{Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges}},
537 + year = {2024},
538 + booktitle = {GEM2 Workshop, ACL 2025},
539 + eprint = {2406.12624},
540 + archivePrefix = {arXiv}
541 +}
542 +
543 +% verified via https://export.arxiv.org/api/query?id_list=2404.13076 on 2026-10-30
544 +@inproceedings{panickssery2024selfpreference,
545 + author = {Arjun Panickssery and Samuel R. Bowman and Shi Feng},
546 + title = {{LLM Evaluators Recognize and Favor Their Own Generations}},
547 + year = {2024},
548 + booktitle = {NeurIPS 2024},
549 + eprint = {2404.13076},
550 + archivePrefix = {arXiv}
551 +}
552 +
553 +% verified via https://export.arxiv.org/api/query?id_list=2507.02825 on 2026-10-30
554 +@misc{zhu2025abc,
555 + author = {Yuxuan Zhu and Tengjun Jin and Yada Pruksachatkun and Andy Zhang and Shu Liu and Sasha Cui and Sayash Kapoor and Shayne Longpre and Kevin Meng and Rebecca Weiss and Fazl Barez and Rahul Gupta and Jwala Dhamala and Jacob Merizian and Mario Giulianelli and Harry Coppock and Cozmin Ududec and Jasjeet Sekhon and Jacob Steinhardt and Antony Kellermann and Sarah Schwettmann and Matei Zaharia and Ion Stoica and Percy Liang and Daniel Kang},
556 + title = {{Establishing Best Practices for Building Rigorous Agentic Benchmarks}},
557 + year = {2025},
558 + howpublished = {arXiv preprint arXiv:2507.02825},
559 + eprint = {2507.02825},
560 + archivePrefix = {arXiv}
561 +}
562 +
563 +% verified via https://export.arxiv.org/api/query?id_list=2510.11977 on 2026-10-30
564 +@misc{kapoor2025hal,
565 + author = {Sayash Kapoor and Benedikt Stroebl and Peter Kirgis and Nitya Nadgir and Zachary S Siegel and Boyi Wei and Tianci Xue and Ziru Chen and Felix Chen and Saiteja Utpala and Franck Ndzomga and Dheeraj Oruganty and Sophie Luskin and Kangheng Liu and Botao Yu and Amit Arora and Dongyoon Hahm and Harsh Trivedi and Huan Sun and Juyong Lee and Tengjun Jin and Yifan Mai and Yifei Zhou and Yuxuan Zhu and Rishi Bommasani and Daniel Kang and Dawn Song and Peter Henderson and Yu Su and Percy Liang and Arvind Narayanan},
566 + title = {{Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation}},
567 + year = {2025},
568 + howpublished = {arXiv preprint arXiv:2510.11977},
569 + eprint = {2510.11977},
570 + archivePrefix = {arXiv}
571 +}
572 +
573 +% verified via https://export.arxiv.org/api/query?id_list=2510.12367 on 2026-10-30
574 +@misc{li2025llmreval,
575 + author = {Rui Li and Jia-Chen Gu and Po-Nien Kung and Heming Xia and Junfeng liu and Xiangwen Kong and Zhifang Sui and Nanyun Peng},
576 + title = {{LLM-REVal: Can We Trust LLM Reviewers Yet?}},
577 + year = {2025},
578 + howpublished = {arXiv preprint arXiv:2510.12367},
579 + eprint = {2510.12367},
580 + archivePrefix = {arXiv}
581 +}
582 +
583 +% verified via https://api.crossref.org/works/10.1518/001872097778543886 on 2026-10-30
584 +@article{parasuraman1997automation,
585 + author = {Parasuraman, Raja and Riley, Victor},
586 + title = {{Humans and Automation: Use, Misuse, Disuse, Abuse}},
587 + year = {1997},
588 + journal = {Human Factors: The Journal of the Human Factors and Ergonomics Society},
589 + volume = {39},
590 + number = {2},
591 + pages = {230--253},
592 + doi = {10.1518/001872097778543886}
593 +}
594 +
595 +% verified via https://api.crossref.org/works/10.1109/3468.844354 on 2026-10-30
596 +@article{parasuraman2000levels,
597 + author = {Parasuraman, R. and Sheridan, T.B. and Wickens, C.D.},
598 + title = {{A model for types and levels of human interaction with automation}},
599 + year = {2000},
600 + journal = {IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans},
601 + volume = {30},
602 + number = {3},
603 + pages = {286--297},
604 + doi = {10.1109/3468.844354}
605 +}
606 +
607 +% verified via https://api.crossref.org/works/10.1136/amiajnl-2011-000089 on 2026-10-30
608 +@article{goddard2012automationbias,
609 + author = {Goddard, Kate and Roudsari, Abdul and Wyatt, Jeremy C},
610 + title = {{Automation bias: a systematic review of frequency, effect mediators, and mitigators}},
611 + year = {2012},
612 + journal = {Journal of the American Medical Informatics Association},
613 + volume = {19},
614 + number = {1},
615 + pages = {121--127},
616 + doi = {10.1136/amiajnl-2011-000089}
617 +}
618 +
619 +% verified via https://api.crossref.org/works/10.1145/3593013.3594087 on 2026-10-30
620 +@inproceedings{lai2023humanai,
621 + author = {Lai, Vivian and Chen, Chacha and Smith-Renner, Alison and Liao, Q. Vera and Tan, Chenhao},
622 + title = {{Towards a Science of Human-AI Decision Making: An Overview of Design Space in Empirical Human-Subject Studies}},
623 + year = {2023},
624 + booktitle = {2023 ACM Conference on Fairness Accountability and Transparency},
625 + pages = {1369--1385},
626 + doi = {10.1145/3593013.3594087},
627 + eprint = {2112.11471},
628 + archivePrefix = {arXiv}
629 +}
630 +
631 +% verified via https://api.crossref.org/works/10.1145/3411764.3445717 on 2026-10-30
632 +@inproceedings{bansal2021whole,
633 + author = {Bansal, Gagan and Wu, Tongshuang and Zhou, Joyce and Fok, Raymond and Nushi, Besmira and Kamar, Ece and Ribeiro, Marco Tulio and Weld, Daniel},
634 + title = {{Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance}},
635 + year = {2021},
636 + booktitle = {Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems},
637 + pages = {1--16},
638 + doi = {10.1145/3411764.3445717}
639 +}
640 +
641 +% verified via https://export.arxiv.org/api/query?id_list=1902.03245 on 2026-10-30
642 +@inproceedings{lubars2019delegability,
643 + author = {Brian Lubars and Chenhao Tan},
644 + title = {{Ask Not What AI Can Do, But What AI Should Do: Towards a Framework of Task Delegability}},
645 + year = {2019},
646 + booktitle = {NeurIPS 2019},
647 + eprint = {1902.03245},
648 + archivePrefix = {arXiv}
649 +}
650 +
651 +% verified via https://api.crossref.org/works/10.1038/s41562-024-02024-1 on 2026-10-30
652 +@article{vaccaro2024combinations,
653 + author = {Vaccaro, Michelle and Almaatouq, Abdullah and Malone, Thomas},
654 + title = {{When combinations of humans and AI are useful: A systematic review and meta-analysis}},
655 + year = {2024},
656 + journal = {Nature Human Behaviour},
657 + volume = {8},
658 + number = {12},
659 + pages = {2293--2303},
660 + doi = {10.1038/s41562-024-02024-1}
661 +}
662 +
663 +% verified via https://api.crossref.org/works/10.1016/0005-1098(83)90046-8 on 2026-10-30
664 +@article{bainbridge1983ironies,
665 + author = {Bainbridge, Lisanne},
666 + title = {{Ironies of automation}},
667 + year = {1983},
668 + journal = {Automatica},
669 + volume = {19},
670 + number = {6},
671 + pages = {775--779},
672 + doi = {10.1016/0005-1098(83)90046-8}
673 +}
674 +
675 +% verified via https://api.crossref.org/works/10.1145/3449287 on 2026-10-30
676 +@article{bucinca2021trust,
677 + author = {Buçinca, Zana and Malaya, Maja Barbara and Gajos, Krzysztof Z.},
678 + title = {{To Trust or to Think}},
679 + year = {2021},
680 + journal = {Proceedings of the ACM on Human-Computer Interaction},
681 + volume = {5},
682 + number = {CSCW1},
683 + pages = {1--21},
684 + doi = {10.1145/3449287}
685 +}
686 +
687 +% verified via https://export.arxiv.org/api/query?id_list=2507.09089 on 2026-10-30
688 +@misc{becker2025metr,
689 + author = {Joel Becker and Nate Rush and Elizabeth Barnes and David Rein},
690 + title = {{Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity}},
691 + year = {2025},
692 + howpublished = {arXiv preprint arXiv:2507.09089},
693 + eprint = {2507.09089},
694 + archivePrefix = {arXiv}
695 +}
696 +
697 +% verified via https://api.crossref.org/works/10.1145/2408776.2408794 on 2026-10-30
698 +@article{dean2013tail,
699 + author = {Dean, Jeffrey and Barroso, Luiz André},
700 + title = {{The tail at scale}},
701 + year = {2013},
702 + journal = {Communications of the ACM},
703 + volume = {56},
704 + number = {2},
705 + pages = {74--80},
706 + doi = {10.1145/2408776.2408794}
707 +}
708 +
709 +% verified via https://api.semanticscholar.org/graph/v1/paper/87c9e34c39dd1fbd470010dc1ea98ba3517f5a2a on 2026-10-30
710 +@inproceedings{gray1986whystop,
711 + author = {Jim Gray},
712 + title = {{Why Do Computers Stop and What Can Be Done About It?}},
713 + year = {1986},
714 + booktitle = {Symposium on Reliability in Distributed Software and Database Systems}
715 +}
716 +
717 +% verified via https://api.crossref.org/works/10.1109/MS.2016.60 on 2026-10-30
718 +@article{basiri2016chaos,
719 + author = {Basiri, Ali and Behnam, Niosha and de Rooij, Ruud and Hochstein, Lorin and Kosewski, Luke and Reynolds, Justin and Rosenthal, Casey},
720 + title = {{Chaos Engineering}},
721 + year = {2016},
722 + journal = {IEEE Software},
723 + volume = {33},
724 + number = {3},
725 + pages = {35--41},
726 + doi = {10.1109/MS.2016.60}
727 +}
728 +
729 +% verified via https://api.crossref.org/works/10.1145/2160718.2160734 on 2026-10-30
730 +@article{helland2012idempotence,
731 + author = {Helland, Pat},
732 + title = {{Idempotence is not a medical condition}},
733 + year = {2012},
734 + journal = {Communications of the ACM},
735 + volume = {55},
736 + number = {5},
737 + pages = {56--65},
738 + doi = {10.1145/2160718.2160734}
739 +}
740 +
741 +% verified via https://api.crossref.org/works/10.1145/3485510 on 2026-10-30
742 +@article{burckhardt2021durablefunctions,
743 + author = {Burckhardt, Sebastian and Gillum, Chris and Justo, David and Kallas, Konstantinos and McMahon, Connor and Meiklejohn, Christopher S.},
744 + title = {{Durable functions: semantics for stateful serverless}},
745 + year = {2021},
746 + journal = {Proceedings of the ACM on Programming Languages},
747 + volume = {5},
748 + number = {OOPSLA},
749 + pages = {1--27},
750 + doi = {10.1145/3485510}
751 +}
752 +
753 +% verified via https://export.arxiv.org/api/query?id_list=2504.13359 on 2026-10-30
754 +@misc{erol2025costofpass,
755 + author = {Mehmet Hamza Erol and Batu El and Mirac Suzgun and Mert Yuksekgonul and James Zou},
756 + title = {{Cost-of-Pass: An Economic Framework for Evaluating Language Models}},
757 + year = {2025},
758 + howpublished = {arXiv preprint arXiv:2504.13359},
759 + eprint = {2504.13359},
760 + archivePrefix = {arXiv}
761 +}
762 +
763 +% verified via https://export.arxiv.org/api/query?id_list=2411.05285 on 2026-10-30
764 +@misc{dong2024agentops,
765 + author = {Liming Dong and Qinghua Lu and Liming Zhu},
766 + title = {{AgentOps: Enabling Observability of LLM Agents}},
767 + year = {2024},
768 + howpublished = {arXiv preprint arXiv:2411.05285},
769 + eprint = {2411.05285},
770 + archivePrefix = {arXiv}
771 +}
772 +
773 +% verified via https://export.arxiv.org/api/query?id_list=2603.29231 on 2026-10-30
774 +@misc{khanal2026reliability,
775 + author = {Aaditya Khanal and Yangyang Tao and Junxiu Zhou},
776 + title = {{Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents}},
777 + year = {2026},
778 + howpublished = {arXiv preprint arXiv:2603.29231},
779 + eprint = {2603.29231},
780 + archivePrefix = {arXiv}
781 +}
782 +
783 +% verified via https://export.arxiv.org/api/query?id_list=2608.14380 on 2026-10-30
784 +@misc{zhuang2026agentrewind,
785 + author = {Yu Zhuang and Kefei Chen and Yitong Duan and Shuxin Zheng and Jian Li and Xu-Yao Zhang},
786 + title = {{AgentRewind: Recoverable Execution for Long-Horizon LLM Agents}},
787 + year = {2026},
788 + howpublished = {arXiv preprint arXiv:2608.14380},
789 + eprint = {2608.14380},
790 + archivePrefix = {arXiv}
791 +}
792 +
793 +% verified via https://api.semanticscholar.org/graph/v1/paper/003d5a65de0ac72daaf105ded903cb3eb88585b3 on 2026-10-30
794 +@techreport{sigelman2010dapper,
795 + author = {B. Sigelman and L. Barroso and M. Burrows and Patrick Stephenson and M. Plakal and Donald Beaver and Saul Jaspan and C. Shanbhag},
796 + title = {{Dapper, a Large-Scale Distributed Systems Tracing Infrastructure}},
797 + year = {2010},
798 + institution = {Google},
799 + number = {dapper-2010-1}
800 +}
801 +
802 +% verified via https://api.crossref.org/works/10.18653/v1/2020.emnlp-main.609 on 2026-10-30
803 +@inproceedings{wadden2020scifact,
804 + author = {Wadden, David and Lin, Shanchuan and Lo, Kyle and Wang, Lucy Lu and van Zuylen, Madeleine and Cohan, Arman and Hajishirzi, Hannaneh},
805 + title = {{Fact or Fiction: Verifying Scientific Claims}},
806 + year = {2020},
807 + booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
808 + pages = {7534--7550},
809 + doi = {10.18653/v1/2020.emnlp-main.609}
810 +}
811 +
812 +% verified via https://api.crossref.org/works/10.18653/v1/N18-1074 on 2026-10-30
813 +@inproceedings{thorne2018fever,
814 + author = {Thorne, James and Vlachos, Andreas and Christodoulopoulos, Christos and Mittal, Arpit},
815 + title = {{FEVER: a Large-scale Dataset for Fact Extraction and VERification}},
816 + year = {2018},
817 + booktitle = {Proceedings of the 2018 Conference of the North American Chapter of
818 + the Association for Computational Linguistics: Human Language
819 + Technologies, Volume 1 (Long Papers)},
820 + pages = {809--819},
821 + doi = {10.18653/v1/N18-1074}
822 +}
823 +
824 +% verified via https://export.arxiv.org/api/query?id_list=2212.08037 on 2026-10-30
825 +@misc{bohnet2022attributedqa,
826 + author = {Bernd Bohnet and Vinh Q. Tran and Pat Verga and Roee Aharoni and Daniel Andor and Livio Baldini Soares and Massimiliano Ciaramita and Jacob Eisenstein and Kuzman Ganchev and Jonathan Herzig and Kai Hui and Tom Kwiatkowski and Ji Ma and Jianmo Ni and Lierni Sestorain Saralegui and Tal Schuster and William W. Cohen and Michael Collins and Dipanjan Das and Donald Metzler and Slav Petrov and Kellie Webster},
827 + title = {{Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models}},
828 + year = {2022},
829 + howpublished = {arXiv preprint arXiv:2212.08037},
830 + eprint = {2212.08037},
831 + archivePrefix = {arXiv}
832 +}
833 +
834 +% verified via https://api.crossref.org/works/10.1038/s41598-023-41032-5 on 2026-10-30
835 +@article{walters2023fabrication,
836 + author = {Walters, William H. and Wilder, Esther Isabelle},
837 + title = {{Fabrication and errors in the bibliographic citations generated by ChatGPT}},
838 + year = {2023},
839 + journal = {Scientific Reports},
840 + volume = {13},
841 + number = {1},
842 + doi = {10.1038/s41598-023-41032-5}
843 +}
844 +
845 +% verified via https://api.crossref.org/works/10.7759/cureus.35179 on 2026-10-30
846 +@article{alkaissi2023hallucinations,
847 + author = {Alkaissi, Hussam and McFarlane, Samy I},
848 + title = {{Artificial Hallucinations in ChatGPT: Implications in Scientific Writing}},
849 + year = {2023},
850 + journal = {Cureus},
851 + doi = {10.7759/cureus.35179}
852 +}
853 +
854 +% verified via https://api.crossref.org/works/10.18653/v1/2023.acl-long.910 on 2026-10-30
855 +@inproceedings{gao2023rarr,
856 + author = {Gao, Luyu and Dai, Zhuyun and Pasupat, Panupong and Chen, Anthony and Chaganty, Arun Tejasvi and Fan, Yicheng and Zhao, Vincent and Lao, Ni and Lee, Hongrae and Juan, Da-Cheng and Guu, Kelvin},
857 + title = {{RARR: Researching and Revising What Language Models Say, Using Language Models}},
858 + year = {2023},
859 + booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
860 + pages = {16477--16508},
861 + doi = {10.18653/v1/2023.acl-long.910}
862 +}
863 +
864 +% verified via https://export.arxiv.org/api/query?id_list=2203.11171 on 2026-10-30
865 +@inproceedings{wang2023selfconsistency,
866 + author = {Xuezhi Wang and Jason Wei and Dale Schuurmans and Quoc Le and Ed Chi and Sharan Narang and Aakanksha Chowdhery and Denny Zhou},
867 + title = {{Self-Consistency Improves Chain of Thought Reasoning in Language Models}},
868 + year = {2022},
869 + booktitle = {ICLR 2023},
870 + eprint = {2203.11171},
871 + archivePrefix = {arXiv}
872 +}
873 +
874 +% verified via https://api.crossref.org/works/10.18653/v1/2023.emnlp-main.741 on 2026-10-30
875 +@inproceedings{min2023factscore,
876 + author = {Min, Sewon and Krishna, Kalpesh and Lyu, Xinxi and Lewis, Mike and Yih, Wen-tau and Koh, Pang and Iyyer, Mohit and Zettlemoyer, Luke and Hajishirzi, Hannaneh},
877 + title = {{FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation}},
878 + year = {2023},
879 + booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
880 + pages = {12076--12100},
881 + doi = {10.18653/v1/2023.emnlp-main.741},
882 + eprint = {2305.14251},
883 + archivePrefix = {arXiv}
884 +}
885 +
886 +% verified via https://export.arxiv.org/api/query?id_list=2310.01798 on 2026-10-30
887 +@inproceedings{huang2024selfcorrect,
888 + author = {Jie Huang and Xinyun Chen and Swaroop Mishra and Huaixiu Steven Zheng and Adams Wei Yu and Xinying Song and Denny Zhou},
889 + title = {{Large Language Models Cannot Self-Correct Reasoning Yet}},
890 + year = {2023},
891 + booktitle = {ICLR 2024},
892 + eprint = {2310.01798},
893 + archivePrefix = {arXiv}
894 +}
895 +
896 +% verified via https://export.arxiv.org/api/query?id_list=2601.18724 on 2026-10-30
897 +@inproceedings{sakai2026hallucitation,
898 + author = {Yusuke Sakai and Hidetaka Kamigaito and Taro Watanabe},
899 + title = {{HalluCitation Matters: Revealing the Impact of Hallucinated References with 300 Hallucinated Papers in ACL Conferences}},
900 + year = {2026},
901 + booktitle = {ACL 2026 (per Semantic Scholar venue record)},
902 + eprint = {2601.18724},
903 + archivePrefix = {arXiv}
904 +}
905 +
906 +% verified via https://export.arxiv.org/api/query?id_list=2605.07723 on 2026-10-30
907 +@misc{zhao2026hallucinationswild,
908 + author = {Zhenyue Zhao and Yihe Wang and Toby Stuart and Mathijs De Vaan and Paul Ginsparg and Yian Yin},
909 + title = {{LLM hallucinations in the wild: Large-scale evidence from non-existent citations}},
910 + year = {2026},
911 + howpublished = {arXiv preprint arXiv:2605.07723},
912 + eprint = {2605.07723},
913 + archivePrefix = {arXiv}
914 +}
915 +
added paper/sections/conclusion.tex +1 −0
@@ -0,0 +1 @@
1 +% conclusion — to be written from verified results
added paper/sections/corpus.tex +61 −0
@@ -0,0 +1,61 @@
1 +\section{Corpus-informed method design}
2 +\label{sec:corpus}
3 +
4 +The protocol compared in arms B--D was derived from a content coding of the instruction files---the \texttt{CLAUDE.md} mandates---that the author had previously issued to Claude Code for earlier working papers in this series. The aim was to learn what a researcher writing such mandates repeatedly asks for and omits, and to turn the omissions into design elements. The exercise motivates the design; it makes no prevalence claim.
5 +
6 +\subsection{Corpus and coding}
7 +
8 +Thirteen files were retrieved from the author's git server on 2026-10-29 and hashed (SHA-256) into a manifest against which every file was re-verified before coding. They contain eleven distinct mandates (WP15, WP18--WP26 and WP28), one supplementary design brief (WP27), and one byte-identical duplicate of the WP15 mandate, which is recorded but counted once. The WP27 brief is a post-implementation description of a simulator rather than an instruction file and is excluded from every ``of 11'' count; WP27's own mandate could not be retrieved (HTTP 404) and nothing is inferred about it. The mandates range from 1,092 to 8,809 words (median 1,966); two (WP15 and WP28) are an order of magnitude longer than the rest, the extra length tracking engineering prescription rather than substantive research rules.
9 +
10 +Coding followed a written codebook. Automated variables (length, headings, phase and gate labels, keyword families) were computed by a script from the file bytes; analyst variables were assigned in a single automated-plus-LLM-analyst pass after reading each document in full, each judgement recorded as \emph{yes}, \emph{partial} or \emph{no} with a verbatim quotation in the same cell so that it can be re-checked against the source. \emph{Partial} means the feature is addressed only indirectly, conditionally or without an operational mechanism. There was no human second coder, so no intercoder reliability is reported and none should be inferred. Table~\ref{tab:corpus} gives the counts.
11 +
12 +\begin{table}[htbp]
13 +\centering
14 +\small
15 +\caption{Feature counts over the eleven distinct mandates (WP27 brief and the WP15 duplicate excluded). ``Partial'' is defined in the codebook as present but indirect, conditional or without an operational mechanism; ``---'' means the category was not used for that feature.}
16 +\label{tab:corpus}
17 +\begin{tabular}{>{\raggedright\arraybackslash}p{2.3in}ccc>{\raggedright\arraybackslash}p{2.05in}}
18 +\toprule
19 +Feature & Yes & Partial & No & Note \\
20 +\midrule
21 +Explicit research question (verbatim) & 10 & --- & 1 & WP22 states objectives, not a question \\
22 +Explicit estimand or hypotheses & 7 & 4 & 0 & H1--H6 lists, theorem queues, primary outcome \\
23 +Assumptions stated in the mandate & 2 & 8 & 1 & most require assumptions without stating them \\
24 +Falsification / null-result conditions & 6 & 4 & 1 & \\
25 +Literature / novelty requirement & 11 & 0 & 0 & universal; usually a ``novelty gate'' \\
26 +Data-access controls & 7 & 4 & 0 & credentials, scraping, privacy, licensing \\
27 +Temporal-information controls & 6 & 1 & 4 & all six data-based mandates; n/a for theory \\
28 +Baseline / benchmark requirement & 8 & 3 & 0 & \\
29 +Phases declared & 10 & --- & 1 & 5 to 13 phases; median 7 \\
30 +Dependency / gate structure & 10 & 1 & 0 & ``partial'' = implicit ordering (WP21) \\
31 +Stopping rules & 2 & 6 & 3 & ``partial'' = document and deliver partial result \\
32 +Recovery / resume instructions & 2 & 6 & 3 & ``partial'' = progress logs, caching, retry limits \\
33 +Provenance requirements & 9 & 2 & 0 & manifests with hashes in WP15, WP28 \\
34 +Review / self-audit stage & 10 & 1 & 0 & results audits, simulated referees, red team \\
35 +AI-use / disclosure statement & 1 & 0 & 10 & WP28 only \\
36 +Budget / resource limits & 1 & 3 & 7 & WP26 asks for API-cost tiers; no numeric cap \\
37 +Integrity prohibitions (``never invent \ldots'') & 11 & 0 & 0 & universal, usually in the opening paragraph \\
38 +Pre-registration mentioned (keyword) & 4 & --- & 7 & no registry or timestamp mechanism \\
39 +Random seeds mentioned (keyword) & 11 & --- & 0 & \\
40 +Statements marked as added or reconstructed & 6 & --- & 5 & \\
41 +\bottomrule
42 +\end{tabular}
43 +\end{table}
44 +
45 +\subsection{What the mandates consistently require}
46 +
47 +Eight patterns recur. (1) Every mandate opens by casting Claude Code as a multidisciplinary research team and immediately prohibits fabrication of data, results, citations, proofs or significance (11 of 11), the most stable element of the corpus. (2) Ten declare ordered phases (five to thirteen; median seven), nine make drafting conditional on validated results, and ten name gates, mostly as numbered acceptance gates. (3) All eleven require a verified literature review or novelty gate before any contribution claim. (4) Ten include audit or self-review as a phase: claim-to-file audits, an adversarial referee report, three simulated referees, or a red-team agent. (5) Nine require traceability of every number to a result file, registry or hashed manifest. (6) All six data-based mandates impose information-set discipline (look-ahead, as-of-date, vintage and chronological-split rules). (7) Eight name baselines. (8) Most provide an honest-failure fallback when work is blocked.
48 +
49 +Four excerpts show the register. The integrity preamble: ``Never fabricate data, results, citations, institutional details, variable definitions, statistical significance, sample sizes, or conclusions. Never write a result in the paper before it has been generated and verified from a saved machine-readable output.'' (\texttt{WP15\_CLAUDE.md}). A phase gate before drafting: ``do not start manuscript drafting until the baseline model is validated, claimed GE equilibria clear markets, principal simulations and exhibits reproduce, bibliography is verified, and theoretical claims have proofs or appropriate qualifications. If blocked, document the issue honestly and deliver a scoped partial result.'' (\texttt{WP18\_CLAUDE.md}). A pre-specified primary outcome: ``Pre-register primary endpoints and hypotheses before running final test evaluations. Primary outcome: out-of-time absolute log-price error on arm's-length residential transactions.'' (\texttt{WP26\_CLAUDE.md}). A declared acceptable null: ``A dead end with a mathematical explanation is better than a fabricated breakthrough. A strong negative result or counterexample may itself become the contribution.'' (\texttt{WP28\_CLAUDE.md}).
50 +
51 +\subsection{What the mandates consistently lack}
52 +
53 +The same documents are silent on mechanism. (1) Gates are declarative. Only WP15 and WP28 attach a gate to something that can fail mechanically---``The release script must fail if: a table references a missing result; a figure is missing source data; LaTeX does not compile; citations are unresolved; \ldots'' (\texttt{WP15\_CLAUDE.md})---and no mandate specifies who or what checks a gate other than the executing agent. (2) Verification is not independent. ``Independent audit'', ``independently checked'', ``independently verified'', ``hostile-referee critique'' and ``three independent review perspectives'' are all to be performed by the same agent; WP28 alone names the tension: ``Record unresolved issues candidly; do not have the same model merely declare its own work accepted.'' (\texttt{WP28\_CLAUDE.md}). (3) Pre-registration is not operational: four mandates ask for it, none specifies a registry, a timestamp, a hash or a freeze. (4) Stopping rules are rare: only WP15, with a dedicated stop-conditions section, and WP28 define when to stop; most others say to continue with feasible work. (5) Recovery is rare: only WP15 (resumable, idempotent, checkpointed jobs) and WP25 require it. (6) Budgets are absent: seven of eleven set no resource limit, only WP26 asks for an API-cost budget, and no mandate states a numeric token, dollar or compute cap. (7) Ten of eleven say nothing about disclosing AI involvement in the paper. (8) Success criteria are not quantified: targets read ``substantially improve'', ``publication-quality'' or ``the standard expected in a top finance journal'', none checkable from the document. (9) The instruction text itself has uncertain provenance: WP23 states that part of its protocol was reconstructed by an assistant after truncation, WP24 and WP26 carry sections marked as added on 2026-10-27, and WP18, WP20 and WP25 carry prefatory repository-mapping notes, so the mandate as retrieved is not always the mandate as issued.
54 +
55 +\subsection{What the corpus cannot establish}
56 +
57 +These documents are instructions: they record what the author asked Claude Code to do, not what was done. The coding therefore provides no evidence about how the earlier working papers were actually executed, whether the instructions were followed, or whether the papers were produced without human assistance; several mandates in fact anticipate human steps such as expert raters and provision of credentials. A mandate that requires a provenance manifest is coded as requiring one; whether the manifest exists is outside the coding. The corpus is also small and researcher-selected---eleven files by one author for one series---and was coded in a single pass without reliability statistics. Its counts motivate design patterns; they do not estimate the prevalence of any practice among autonomous-research mandates in general, and no inferential statistic is computed from them. Questions about execution require run-level evidence, which the experiment that follows is designed to produce.
58 +
59 +\subsection{From absences to design elements}
60 +
61 +The corpus shows a stable rhetorical protocol (role, integrity rules, phases, gates, audits, traceability) and an equally stable absence of mechanisms. Rather than add prose rules, arms B--D turn each absence into something checkable. Uncertain provenance of the instruction (absence 9) and non-operational pre-registration (absence 3) become a \emph{hashed, frozen mandate}: the instruction injected into each run is fixed and hashed before execution, so that what the executor received is known exactly. Declarative gates (absence 1) become \emph{executable gates} enforced by the controller in arms C and D, which can fail mechanically rather than by the agent's own judgement. Self-performed audits (absence 2) become \emph{external evaluators}: hidden deterministic oracles on a separate host, and in arm D a review-and-repair session separate from the executing session, so that no model merely declares its own work accepted. Missing budgets and stopping rules (absences 4 and 6) become \emph{explicit budgets and stop rules} in the protocol. The unrecorded human steps that several mandates anticipate become an \emph{intervention ledger} in which any human intervention during a run is recorded. And the audit phase that the mandates ask the same agent to perform becomes the \emph{separate review} that distinguishes arm D from arm C. Arm A, the open instruction, receives the task without any of these mechanisms, and so serves as the control against which their contribution is measured.
added paper/sections/design.tex +1 −0
@@ -0,0 +1 @@
1 +% design — to be written from verified results
added paper/sections/evaluation.tex +1 −0
@@ -0,0 +1 @@
1 +% evaluation — to be written from verified results
added paper/sections/framework.tex +48 −0
Large diff (48 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/implementation.tex +39 −0
Large diff (39 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/introduction.tex +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/limitations.tex +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/literature.tex +90 −0
Large diff (90 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/recovery.tex +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/reflexive.tex +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/results.tex +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/titlepage.tex +77 −0
Large diff (77 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/titlepage_anon.tex +53 −0
Large diff (53 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/transfer.tex +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added paper/uq_logo.jpg +0 −0

Binary file not shown.

modified protocol/amendments.csv +1 −0
Large diff (3 lines) is not loaded by default to keep the page fast.Load diff
added protocol/arms/arm_A_long.md +13 −0
Large diff (13 lines) is not loaded by default to keep the page fast.Load diff
added protocol/preregistration.md +33 −0
Large diff (33 lines) is not loaded by default to keep the page fast.Load diff
added reports/evaluator_validation.md +40 −0
Large diff (40 lines) is not loaded by default to keep the page fast.Load diff
modified reports/phase_status.md +5 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added research/novelty_matrix.csv +19 −0
Large diff (19 lines) is not loaded by default to keep the page fast.Load diff
added research/source_notes.md +102 −0
Large diff (102 lines) is not loaded by default to keep the page fast.Load diff
added results/analysis_data/build_summary.json +9 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added results/estimates/development_arm_table.csv +5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added results/estimates/development_contrasts.csv +25 −0
Large diff (25 lines) is not loaded by default to keep the page fast.Load diff
added results/estimates/development_family_table.csv +5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added results/figures/fig_cost_development.pdf +0 −0

Binary file not shown.

added results/figures/fig_family_development.pdf +0 −0

Binary file not shown.

added results/figures/fig_primary_development.pdf +0 −0

Binary file not shown.

added results/figures/fig_quality_development.pdf +0 −0

Binary file not shown.

added results/tables/tab_accounting.tex +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added results/tables/tab_arms_development.tex +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added results/tables/tab_contrasts_development.tex +30 −0
Large diff (30 lines) is not loaded by default to keep the page fast.Load diff
added results/tables/tab_cost_development.tex +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added results/tables/tab_family_development.tex +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0001/artifact_manifest.json +10 −0
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0001/behaviour_audit.json +5 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0001/contract_receipt.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0001/ctl/prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0001/ctl/reviewer_prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0001/ctl/reviewer_system.md +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0001/ctl/system_append.md +16 −0
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0001/evaluation_receipt.json +98 −0
Large diff (99 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0001/out/analysis.py +107 −0
Large diff (107 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0001/out/claims.json +5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0001/out/report.md +31 −0
Large diff (31 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0001/out/results.json +34 −0
Large diff (34 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0001/prepare.json +21 −0
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0001/sessions/e87c6a93-b2a9-49ef-9e9c-af14a9a542ea.jsonl +44 −0
Large diff (44 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0001/system_append.md +16 −0
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/artifact_manifest.json +21 −0
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/behaviour_audit.json +10 −0
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/contract_receipt.json +147 −0
Large diff (148 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/ctl/prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/ctl/reviewer_prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/ctl/reviewer_system.md +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/ctl/system_append.md +59 −0
Large diff (59 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/evaluation_receipt.json +98 −0
Large diff (99 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/out/analysis.py +137 −0
Large diff (137 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/out/claims.json +6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/out/exclusions.csv +2 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/out/report.md +26 −0
Large diff (26 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/out/results.json +48 −0
Large diff (48 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/out/review/review.json +20 −0
Large diff (20 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/out/state/checks.md +16 −0
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/out/state/contract.md +13 −0
Large diff (13 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/out/state/data_audit.md +23 −0
Large diff (23 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/out/state/gate_receipts/20261030T034924.971935Z.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/out/state/gate_receipts/20261030T034949.861067Z.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/out/state/gate_receipts/20261030T034953.949373Z.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/out/state/gate_receipts/20261030T035000.780240Z.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/out/state/log.md +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/out/state/phases.json +51 −0
Large diff (52 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/prepare.json +21 −0
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/sessions/5aec3cd4-37d1-4c46-944a-1b2a916d9f46.jsonl +38 −0
Large diff (38 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/sessions/6b68b46d-6ed4-47a3-bce3-ea897249c271.jsonl +81 −0
Large diff (81 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0002/system_append.md +59 −0
Large diff (59 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/artifact_manifest.json +20 −0
Large diff (21 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/behaviour_audit.json +5 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/contract_receipt.json +147 −0
Large diff (148 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/ctl/prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/ctl/reviewer_prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/ctl/reviewer_system.md +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/ctl/system_append.md +56 −0
Large diff (56 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/evaluation_receipt.json +98 −0
Large diff (99 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/out/analysis.py +115 −0
Large diff (115 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/out/claims.json +6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/out/exclusions.csv +2 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/out/report.md +26 −0
Large diff (26 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/out/results.json +38 −0
Large diff (39 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/out/state/checks.md +9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/out/state/contract.md +6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/out/state/data_audit.md +11 −0
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/out/state/gate_receipts/20261030T034911.071067Z.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/out/state/gate_receipts/20261030T034917.251699Z.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/out/state/gate_receipts/20261030T034920.137046Z.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/out/state/gate_receipts/20261030T034927.073939Z.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/out/state/log.md +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/out/state/phases.json +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/prepare.json +21 −0
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/sessions/18beecef-57d9-42ce-8b7a-703b9a4e0513.jsonl +72 −0
Large diff (72 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0003/system_append.md +56 −0
Large diff (56 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0004/artifact_manifest.json +16 −0
Large diff (17 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0004/behaviour_audit.json +5 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0004/contract_receipt.json +147 −0
Large diff (148 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0004/ctl/prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0004/ctl/reviewer_prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0004/ctl/reviewer_system.md +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0004/ctl/system_append.md +51 −0
Large diff (51 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0004/evaluation_receipt.json +98 −0
Large diff (99 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0004/out/analysis.py +113 −0
Large diff (113 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0004/out/claims.json +5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0004/out/exclusions.csv +2 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0004/out/report.md +24 −0
Large diff (24 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0004/out/results.json +39 −0
Large diff (39 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0004/out/state/checks.md +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0004/out/state/contract.md +6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0004/out/state/data_audit.md +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0004/out/state/log.md +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0004/out/state/phases.json +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0004/prepare.json +21 −0
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0004/sessions/2b91fd6f-c168-4265-a0f2-58c4fe95187d.jsonl +49 −0
Large diff (49 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0004/system_append.md +51 −0
Large diff (51 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/artifact_manifest.json +23 −0
Large diff (24 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/behaviour_audit.json +22 −0
Large diff (23 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/contract_receipt.json +162 −0
Large diff (163 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/ctl/prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/ctl/reviewer_prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/ctl/reviewer_system.md +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/ctl/system_append.md +59 −0
Large diff (59 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/evaluation_receipt.json +154 −0
Large diff (155 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/out/analysis.py +77 −0
Large diff (77 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/out/analytic.csv +395 −0
Large diff (395 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/out/claims.json +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/out/exclusions.csv +4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/out/report.md +30 −0
Large diff (30 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/out/results.json +32 −0
Large diff (33 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/out/review/review.json +6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/out/state/checks.md +15 −0
Large diff (15 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/out/state/contract.md +6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/out/state/data_audit.md +12 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/out/state/gate_receipts/20261030T034838.835406Z.json +160 −0
Large diff (161 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/out/state/gate_receipts/20261030T034903.489772Z.json +160 −0
Large diff (161 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/out/state/gate_receipts/20261030T034903.791756Z.json +160 −0
Large diff (161 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/out/state/gate_receipts/20261030T034908.120415Z.json +160 −0
Large diff (161 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/out/state/gate_receipts/20261030T034917.373222Z.json +160 −0
Large diff (161 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/out/state/log.md +6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/out/state/phases.json +52 −0
Large diff (53 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/prepare.json +20 −0
Large diff (21 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/sessions/1ef94dd9-9edb-49f1-a8b9-f5d66c09266d.jsonl +75 −0
Large diff (75 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/sessions/221ac842-a9ef-4067-a050-74bc9a1bcae9.jsonl +40 −0
Large diff (40 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0005/system_append.md +59 −0
Large diff (59 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/artifact_manifest.json +21 −0
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/behaviour_audit.json +5 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/contract_receipt.json +162 −0
Large diff (163 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/ctl/prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/ctl/reviewer_prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/ctl/reviewer_system.md +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/ctl/system_append.md +56 −0
Large diff (56 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/evaluation_receipt.json +155 −0
Large diff (156 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/out/analysis.py +92 −0
Large diff (92 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/out/analytic.csv +395 −0
Large diff (395 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/out/claims.json +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/out/exclusions.csv +5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/out/report.md +31 −0
Large diff (31 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/out/results.json +34 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/out/state/checks.md +11 −0
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/out/state/contract.md +6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/out/state/data_audit.md +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/out/state/gate_receipts/20261030T034914.352198Z.json +160 −0
Large diff (161 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/out/state/gate_receipts/20261030T034919.593424Z.json +160 −0
Large diff (161 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/out/state/gate_receipts/20261030T034923.408413Z.json +160 −0
Large diff (161 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/out/state/gate_receipts/20261030T034932.641511Z.json +160 −0
Large diff (161 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/out/state/log.md +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/out/state/phases.json +50 −0
Large diff (51 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/prepare.json +20 −0
Large diff (21 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/sessions/9f7cd1a8-e274-4b5d-a059-7aaf1e0f53ea.jsonl +84 −0
Large diff (84 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0006/system_append.md +56 −0
Large diff (56 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0007/artifact_manifest.json +17 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0007/behaviour_audit.json +5 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0007/contract_receipt.json +162 −0
Large diff (163 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0007/ctl/prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0007/ctl/reviewer_prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0007/ctl/reviewer_system.md +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0007/ctl/system_append.md +51 −0
Large diff (51 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0007/evaluation_receipt.json +154 −0
Large diff (155 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0007/out/analysis.py +85 −0
Large diff (85 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0007/out/analytic.csv +395 −0
Large diff (395 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0007/out/claims.json +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0007/out/exclusions.csv +4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0007/out/report.md +37 −0
Large diff (37 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0007/out/results.json +37 −0
Large diff (37 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0007/out/state/checks.md +19 −0
Large diff (19 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0007/out/state/contract.md +6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0007/out/state/data_audit.md +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0007/out/state/log.md +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0007/out/state/phases.json +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0007/prepare.json +20 −0
Large diff (21 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0007/sessions/82946b6b-ee34-4eb6-b743-518f2c4f1830.jsonl +65 −0
Large diff (65 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0007/system_append.md +51 −0
Large diff (51 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0008/artifact_manifest.json +12 −0
Large diff (13 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0008/behaviour_audit.json +5 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0008/contract_receipt.json +157 −0
Large diff (158 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0008/ctl/prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0008/ctl/reviewer_prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0008/ctl/reviewer_system.md +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0008/ctl/system_append.md +16 −0
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0008/evaluation_receipt.json +155 −0
Large diff (156 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0008/out/analysis.py +93 −0
Large diff (93 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0008/out/analytic.csv +395 −0
Large diff (395 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0008/out/claims.json +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0008/out/exclusions.csv +5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0008/out/report.md +37 −0
Large diff (37 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0008/out/results.json +30 −0
Large diff (31 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0008/prepare.json +20 −0
Large diff (21 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0008/sessions/66bc2d76-2305-4e8b-b80f-96b04df3ab15.jsonl +51 −0
Large diff (51 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0008/system_append.md +16 −0
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/artifact_manifest.json +20 −0
Large diff (21 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/behaviour_audit.json +18 −0
Large diff (19 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/contract_receipt.json +177 −0
Large diff (178 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/ctl/prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/ctl/reviewer_prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/ctl/reviewer_system.md +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/ctl/system_append.md +56 −0
Large diff (56 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/evaluation_receipt.json +124 −0
Large diff (125 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/out/analysis.py +92 −0
Large diff (92 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/out/claims.json +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/out/exclusions.csv +4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/out/report.md +35 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/out/results.json +28 −0
Large diff (29 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/out/state/checks.md +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/out/state/contract.md +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/out/state/data_audit.md +9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/out/state/gate_receipts/20261030T034924.898523Z.json +175 −0
Large diff (176 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/out/state/gate_receipts/20261030T034946.701287Z.json +175 −0
Large diff (176 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/out/state/gate_receipts/20261030T034950.623301Z.json +175 −0
Large diff (176 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/out/state/gate_receipts/20261030T034959.520687Z.json +175 −0
Large diff (176 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/out/state/log.md +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/out/state/phases.json +52 −0
Large diff (53 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/prepare.json +20 −0
Large diff (21 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/sessions/6d36a827-c755-489d-ad7b-46c3dd066c8c.jsonl +54 −0
Large diff (54 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0009/system_append.md +56 −0
Large diff (56 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0010/artifact_manifest.json +16 −0
Large diff (17 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0010/behaviour_audit.json +5 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0010/contract_receipt.json +177 −0
Large diff (178 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0010/ctl/prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0010/ctl/reviewer_prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0010/ctl/reviewer_system.md +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0010/ctl/system_append.md +51 −0
Large diff (51 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0010/evaluation_receipt.json +124 −0
Large diff (125 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0010/out/analysis.py +94 −0
Large diff (94 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0010/out/claims.json +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0010/out/exclusions.csv +4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0010/out/report.md +36 −0
Large diff (36 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0010/out/results.json +28 −0
Large diff (29 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0010/out/state/checks.md +11 −0
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0010/out/state/contract.md +6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0010/out/state/data_audit.md +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0010/out/state/log.md +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0010/out/state/phases.json +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0010/prepare.json +20 −0
Large diff (21 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0010/sessions/c148a845-7eb1-46e9-b5b5-af7c3477b616.jsonl +60 −0
Large diff (60 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0010/system_append.md +51 −0
Large diff (51 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0011/artifact_manifest.json +10 −0
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0011/behaviour_audit.json +5 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0011/contract_receipt.json +167 −0
Large diff (168 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0011/ctl/prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0011/ctl/reviewer_prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0011/ctl/reviewer_system.md +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0011/ctl/system_append.md +16 −0
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0011/evaluation_receipt.json +124 −0
Large diff (125 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0011/out/analysis.py +79 −0
Large diff (79 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0011/out/claims.json +6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0011/out/report.md +40 −0
Large diff (40 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0011/out/results.json +28 −0
Large diff (29 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0011/prepare.json +20 −0
Large diff (21 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0011/sessions/4d3727e0-1c22-4f13-b5a0-87bf057ab979.jsonl +42 −0
Large diff (42 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0011/system_append.md +16 −0
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/artifact_manifest.json +21 −0
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/behaviour_audit.json +18 −0
Large diff (19 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/contract_receipt.json +177 −0
Large diff (178 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/ctl/prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/ctl/reviewer_prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/ctl/reviewer_system.md +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/ctl/system_append.md +59 −0
Large diff (59 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/evaluation_receipt.json +124 −0
Large diff (125 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/out/analysis.py +75 −0
Large diff (75 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/out/claims.json +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/out/exclusions.csv +4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/out/report.md +30 −0
Large diff (30 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/out/results.json +26 −0
Large diff (27 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/out/review/review.json +9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/out/state/checks.md +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/out/state/contract.md +6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/out/state/data_audit.md +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/out/state/gate_receipts/20261030T035005.608398Z.json +175 −0
Large diff (176 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/out/state/gate_receipts/20261030T035029.001461Z.json +175 −0
Large diff (176 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/out/state/gate_receipts/20261030T035032.780626Z.json +175 −0
Large diff (176 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/out/state/gate_receipts/20261030T035041.914071Z.json +175 −0
Large diff (176 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/out/state/log.md +6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/out/state/phases.json +50 −0
Large diff (51 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/prepare.json +20 −0
Large diff (21 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/sessions/29963ab3-b996-45cd-a99c-907f85d31074.jsonl +35 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/sessions/6847dcee-4caa-4fe1-9fee-594b975c8e6e.jsonl +63 −0
Large diff (63 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0012/system_append.md +59 −0
Large diff (59 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/artifact_manifest.json +23 −0
Large diff (24 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/behaviour_audit.json +5 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/contract_receipt.json +212 −0
Large diff (213 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/ctl/prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/ctl/reviewer_prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/ctl/reviewer_system.md +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/ctl/system_append.md +56 −0
Large diff (56 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/evaluation_receipt.json +168 −0
Large diff (169 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/out/analysis.py +144 −0
Large diff (144 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/out/claims.json +6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/out/exclusions.csv +4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/out/figure1.png +0 −0

Binary file not shown.

added runs/pilot/RP0013/out/report.md +51 −0
Large diff (51 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/out/results.json +58 −0
Large diff (59 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/out/state/checks.md +12 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/out/state/contract.md +6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/out/state/data_audit.md +16 −0
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/out/state/gate_receipts/20261030T035124.455882Z.json +210 −0
Large diff (211 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/out/state/gate_receipts/20261030T035158.231150Z.json +210 −0
Large diff (211 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/out/state/gate_receipts/20261030T035206.144438Z.json +210 −0
Large diff (211 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/out/state/gate_receipts/20261030T035210.136231Z.json +210 −0
Large diff (211 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/out/state/gate_receipts/20261030T035219.945910Z.json +210 −0
Large diff (211 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/out/state/log.md +5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/out/state/phases.json +52 −0
Large diff (53 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/out/table1.csv +5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/prepare.json +22 −0
Large diff (23 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/sessions/57602e14-17cc-47c9-884e-1967d9060309.jsonl +97 −0
Large diff (97 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0013/system_append.md +56 −0
Large diff (56 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0014/prepare.json +22 −0
Large diff (23 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0014/system_append.md +59 −0
Large diff (59 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0015/artifact_manifest.json +13 −0
Large diff (14 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0015/behaviour_audit.json +5 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0015/contract_receipt.json +202 −0
Large diff (203 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0015/ctl/prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0015/ctl/reviewer_prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0015/ctl/reviewer_system.md +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0015/ctl/system_append.md +16 −0
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0015/evaluation_receipt.json +168 −0
Large diff (169 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0015/out/analysis.py +124 −0
Large diff (124 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0015/out/claims.json +6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0015/out/figure1.png +0 −0

Binary file not shown.

added runs/pilot/RP0015/out/report.md +55 −0
Large diff (55 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0015/out/results.json +49 −0
Large diff (50 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0015/out/sensitivity.json +45 −0
Large diff (46 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0015/out/table1.csv +5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0015/prepare.json +22 −0
Large diff (23 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0015/sessions/28baae04-29de-4b30-a556-82e331248442.jsonl +76 −0
Large diff (76 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0015/system_append.md +16 −0
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0016/artifact_manifest.json +18 −0
Large diff (19 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0016/behaviour_audit.json +5 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0016/contract_receipt.json +212 −0
Large diff (213 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0016/ctl/prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0016/ctl/reviewer_prompt.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0016/ctl/reviewer_system.md +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0016/ctl/system_append.md +51 −0
Large diff (51 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0016/evaluation_receipt.json +168 −0
Large diff (169 lines) is not loaded by default to keep the page fast.Load diff
added runs/pilot/RP0016/out/analysis.py +67 −0
Large diff (67 lines) is not loaded by default to keep the page fast.Load diff

Diff truncated — file too large.

Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere