# WP29 — Source notes for the literature review

Scope: eight streams, 84 verified references (literature_matrix.csv, references_verified.bib), plus a novelty matrix of the closest systems (novelty_matrix.csv). Every reference was verified on 2026-10-30 (UTC) against a primary metadata source — Crossref (api.crossref.org/works/<DOI>), the arXiv API (export.arxiv.org/api/query?id_list=), Semantic Scholar (api.semanticscholar.org/graph/v1/paper/...) and, for ICLR/NeurIPS acceptance, OpenReview (api2.openreview.net) — and every DOI was checked to resolve at doi.org (all returned HTTP 302). Raw API responses are kept in verify_cache/; build_literature_matrix.py regenerates the CSV and the BibTeX from those cached records only. "Key claim" sentences are drawn from the abstracts retrieved through those APIs (or PubMed for three journal abstracts); where I did not read enough to summarize safely the matrix says n/a (one entry: the Nature 2026 version of The AI Scientist, whose abstract sits behind the publisher's login — the arXiv 2024 abstract is used for the system description instead).

Rule applied for peer_reviewed: "yes" only when a journal/proceedings record exists in Crossref, Semantic Scholar's publicationVenue, OpenReview, or the ICLR proceedings site; "no" for arXiv-only items; "unknown" for editorial/bulletin venues (Cortex editorial, American Scientist, CACM practice articles, ACM SIGIR Forum, Cureus editorial, the NASEM consensus report, and Gray 1986 whose only record is a Semantic Scholar/DBLP conference entry).

Dropped as unverifiable within the allowed sources (not cited): Oppenheimer, Ganapathi & Patterson (2003, USITS) and Yuan et al. (2014, OSDI) — no Crossref record and Semantic Scholar search was rate-limited (HTTP 429) on three attempts; Linardon et al. (2025, JMIR Mental Health, DOI 10.2196/80371) was verified but left out as redundant with Walters & Wilder; SWE-bench Verified is a vendor blog post with no citable metadata record and is discussed only through Zhu et al. (2025), who audit it. Two candidate IDs I had wrong were corrected during verification (ChemCrow DOI ends in -00832-8, not -x; DiscoveryBench is arXiv:2407.01725, not 2407.01214).

# Stream 1 — Autonomous scientific discovery and agentic research systems (14)

lu2024aiscientist / lu2026aiscientist. The AI Scientist (arXiv Aug. 2024; the line of work is now published in Nature as "Towards end-to-end automation of AI research", DOI 10.1038/s41586-026-10265-5) chains idea generation, code editing, experiment execution, plotting, paper writing and a simulated review; the preprint reports less than $15 per paper and an automated reviewer with near-human agreement on scores. Design caveats for WP29: a single pipeline with no control arm, human-written code templates, and an in-system reviewer (self-preference risk; see Panickssery et al.). Independent audits (Beel et al. 2025; Luo, Kasirzadeh & Shah 2025) found novelty misclassification, execution failures and leakage/metric misuse. I did not retrieve the Nature abstract, so no headline number is attributed to the 2026 version.

yamada2025aiscientistv2. Removes the template dependence, uses a progressive agentic tree search managed by an experiment-manager agent and a VLM feedback loop for figures. Evidence of validity is one ICLR-2025-workshop submission (of three) scoring above the average acceptance threshold; n = 3 manuscripts, no uncertainty quantification, cost not given in the abstract. Not peer reviewed as a system paper.

schmidgall2025agentlab. Agent Laboratory (Findings of EMNLP 2025) stages literature review, experimentation and report writing and lets humans give feedback at each stage. Findings: o1-preview gave the best outcomes, human feedback at each stage significantly improved quality, and expenses were 84% lower than prior autonomous methods. Units are human survey ratings and automated review scores; the number of raters is modest and the "quality" construct is reviewer score, not verified scientific validity. This is the closest evidence that structure (staging + feedback) matters, but the manipulated factor is human co-piloting, not the instruction protocol.

boiko2023coscientist. Nature 2023. GPT-4 agent with web/docs search, code execution and lab-automation tools executes six chemistry tasks including successful Pd-catalysed cross-coupling optimisation. The oracle is physical; humans supervise execution. Establishes feasibility of tool-using autonomous experimentation, not a comparison of protocols.

bran2024chemcrow. Nature Machine Intelligence 2024. 18 tools + GPT-4; syntheses of an insect repellent and three organocatalysts; evaluation by both LLM and expert assessment — an early acknowledgement that LLM-only evaluation is insufficient.

ifargan2025datatopaper. NEJM AI 2025 (arXiv 2024). data-to-paper walks LLM agents from annotated data to manuscripts while programmatically backtracing information flow so humans can verify every number; autopilot vs copilot modes. Closest prior to WP29's "persistent evidence records", but it is a single pipeline (no arm without traceability) and verification is human, not an executable gate.

gottweis2026coscientist. Nature 2026 (arXiv Feb. 2025). Gemini-based multi-agent generate/critique/rank/evolve loop with tournament evolution and test-time-compute scaling for hypothesis generation; critique is in-system, validation is downstream wet-lab work by collaborators. Scope is hypothesis generation, not executed computational research.

swanson2025virtuallab. Nature 2025. LLM principal-investigator agent runs meetings with specialist agents and a critic; humans guide and run wet-lab validation of designed nanobodies. Shows a human-oversight design point; no controlled comparison.

mitchener2025kosmos. arXiv Nov. 2025. Runs up to 12 h, ~200 rollouts, with a structured world model shared between analysis and literature agents. Important for long-horizon coherence; accuracy is judged by expert reading of reports; no executable validity gates.

luo2025pitfalls. arXiv Sept. 2025 (NeurIPS 2025 AI-for-Science workshop spotlight per the conference site; counted as not peer reviewed). Defines four failure modes — inappropriate benchmark selection, data leakage, metric misuse, post-hoc selection bias — and finds several in two open-source AI scientist systems via controlled experiments. Directly motivates WP29's gates; caveat: two systems, authors' own rubric.

beel2025sakana. ACM SIGIR Forum 2025 (bulletin; editorial review status unknown). Independent re-run of the AI Scientist: keyword-only literature search, ideas wrongly classified as novel, experiment failures. Small-n qualitative audit.

luo2026xscientist. arXiv July 2026. Git-like protocol: typed content-addressed research objects, immutable checkpoints, claim-evidence closure, authority-aware gates; the authors stress a passing integrity check is not scientific truth. Conceptually the nearest neighbour to WP29's protocol + evidence records, but it is a protocol artifact/implementation, not an experiment.

ding2026verificationgap. arXiv June 2026 survey: 125 screened, 35 included, 26 full-text coded on seven audit dimensions; code release is common but reproducibility-grade and claim-verification artifacts are rare. Provides the vocabulary ("verification gap") and audit dimensions WP29 reports against.

# Stream 2 — Language agents for data analysis and scientific programming (9)

chen2025scienceagentbench (ICLR 2025 poster; OpenReview record). 102 tasks from 44 papers, expert-validated, program-file outputs, execution-based metrics plus cost, two contamination mitigations. Best agent 32.4% (34.3% with expert knowledge); o1-preview 42.2% at >10x cost; three attempts per task. The explicit call to assess individual tasks "before making bold claims on end-to-end automation" is WP29's stance.

majumder2025discoverybench (ICLR 2025). 264 real + 903 synthetic tasks; each task = dataset + metadata + discovery goal; hypothesis-level grading. I did not read the headline success rate, so none is quoted.

jing2025dsbench (ICLR 2025). 466 analysis + 74 modeling tasks from Eloquence/Kaggle with long contexts; realism argument; headline rate not quoted.

huang2024mlagentbench (ICML 2024). 13 experimentation tasks; ReAct agent; Claude 3 Opus best success rate. Small task count; success is metric improvement.

gu2024blade (EMNLP 2024). 12 datasets/questions; ground truth from independent expert analyses; evaluates analysis decisions under multiple valid paths — the same oracle problem WP29's rubrics face.

jimenez2024swebench (ICLR 2024). 2,294 GitHub issue/PR problems; test-based grading. Zhu et al. (2025) later show SWE-bench Verified's tests are insufficient — a caution for any test-as-oracle design.

chan2025mlebench (ICLR 2025; proceedings page verified). 75 Kaggle competitions, human leaderboard baselines; best setup medals in 16.9%; resource-scaling and contamination analyses.

siegel2024corebench (TMLR per Semantic Scholar). 270 reproduction tasks from 90 papers in three disciplines; links agent evaluation to computational reproducibility.

starace2025paperbench (ICML 2025). 20 ICML papers, 8,316 rubric items co-developed with authors, LLM judge validated on a separate judge benchmark — the closest precedent for validated rubric oracles.

# Stream 3 — Workflows, provenance, computational reproducibility (10)

koster2012snakemake (Bioinformatics 2012) and ditommaso2017nextflow (Nature Biotechnology 2017, correspondence) are the canonical declarative/dataflow workflow engines; both argue that reproducibility comes from re-executable, environment-pinned workflow definitions. missier2013prov (EDBT 2013) summarises the W3C PROV family (PROV-DM/O/N/constraints) — the vocabulary (entity/activity/agent) that WP29's evidence records instantiate. chirigati2016reprozip (SIGMOD 2016 demo) captures execution provenance automatically into a re-executable package, i.e., evidence captured by the system rather than self-reported by the agent.

sandve2013tensimplerules (PLoS Comput Biol 2013): rules include tracking how every result was produced, avoiding manual data manipulation, archiving program versions and seeds, and public access to scripts and results — WP29 turns these into machine-checked gates. peng2011reproducible (Science 2011) frames reproducibility as the minimum standard when replication is impossible. nasem2019reproducibility (NASEM 2019) fixes the definitions: reproducibility = same data/code/conditions, replicability = new data; WP29 uses these.

stodden2018journalpolicy (PNAS 2018): random sample of 204 Science papers under a data/code-on-request policy; artifacts obtained for 44%, findings reproduced for 26% — policy without executable verification fails. trisovic2022codequality (Scientific Data 2022): >9,000 R files; 74% failed initially, 56% after cleaning. krafczyk2021reproduction (Phil Trans A 2021): three principles (transparency, re-executability, determinism), 12 guidelines, Reproduction Package. Caveats: Stodden's sample is one journal/one policy; Trisovic's failure definition is "any error on clean re-execution".

# Stream 4 — Preregistration, registered reports, specification discipline (10)

nosek2018preregistration (PNAS 2018): preregistration separates prediction from postdiction. chambers2013registeredreports (Cortex 2013 editorial) and chambers2022registeredreports (Nature Human Behaviour 2022 review): protocols reviewed and pre-accepted before data; early evidence they work; not a universal cure. simmons2011falsepositive (Psychological Science 2011): simulations and two experiments show researcher degrees of freedom make false positives easy; six author requirements + four reviewer guidelines. gelman2014statisticalcrisis (American Scientist 2014): garden of forking paths — data-dependent choices inflate false positives without intent. kerr1998harking: presenting post hoc hypotheses as a priori. munafo2017manifesto and nosek2015top: the measures and TOP transparency tiers. scheel2021excess (AMPPS 2021): 96% positive first hypotheses in 152 standard papers vs 44% in 71 Registered Reports — the clearest quantitative effect of specification-before-results, with the caveat that RR authors self-select. osc2015reproducibility (Science 2015): 100 replications, roughly one-third to one-half of findings replicated depending on criterion.

Relevance: WP29's "protocol" arm is, functionally, a preregistration imposed on the agent; the "open instruction" arm is the garden of forking paths with an agent walking it.

# Stream 5 — Agent evaluation, contamination, evaluator validity (10)

liu2024agentbench (ICLR 2024): eight environments; instruction-following and long-horizon reasoning are the main failure causes. yao2024taubench (arXiv 2024; venue unverified): state-based outcome checking and pass^k; gpt-4o <50% success, pass^8 <25% retail — reliability over repeated runs is a separate quantity from success rate. kapoor2024agentsthatmatter (TMLR per Semantic Scholar): cost must be reported alongside accuracy; holdouts are often inadequate; model- vs downstream-developer needs conflated. kapoor2025hal (arXiv Oct. 2025): standardized harness, 21,730 rollouts, ~$40,000. zhu2025abc (arXiv July 2025): task/outcome-validity flaws (SWE-bench Verified insufficient tests; TAU-bench counts empty responses as success) mis-estimate performance by up to 100% relative; checklist cut overestimation on CVE-Bench by 33%. sainz2023contamination (Findings EMNLP 2023): contamination levels and the call to measure per benchmark.

zheng2023mtbench (NeurIPS 2023 D&B): LLM judges have position, verbosity and self-enhancement biases; GPT-4 reaches >80% agreement with humans, equal to human-human. thakur2025judgingjudges (GEM2 workshop 2025): even the best judges lag inter-human agreement; scores can differ by up to 5 points. panickssery2024selfpreference (NeurIPS 2024): self-recognition correlates linearly with self-preference. li2025llmreval (arXiv Oct. 2025): simulated research-and-review loop with human annotation shows pronounced misalignment between LLM reviewers and humans. Together these justify WP29's protected, executor-separated oracles and its refusal to let the executor's own model grade itself.

# Stream 6 — Human-AI delegation, automation bias, intervention measurement (10)

parasuraman1997automation (use/misuse/disuse/abuse) and parasuraman2000levels (four function classes, continuum of levels) give the vocabulary for placing WP29's arms. goddard2012automationbias (JAMIA 2012): systematic review of 74 studies; bias mediated by cognitive style, DSS design and experience. bainbridge1983ironies: automation leaves humans the hardest residual tasks. bansal2021whole (CHI 2021): explanations raise acceptance regardless of correctness. bucinca2021trust (CSCW 2021, N = 199): cognitive forcing reduces overreliance — the design principle behind requiring reviewers to execute checks rather than read agent summaries. lubars2019delegability (NeurIPS 2019): strong preference for machine-in-the-loop over full AI control; trust dominates. lai2023humanai (FAccT 2023): design space of empirical human-AI decision studies. vaccaro2024combinations (Nature Human Behaviour 2024): 106 experiments, 370 effect sizes; combinations on average below the best of human or AI alone, with gains in creation tasks. becker2025metr (arXiv July 2025): RCT, 16 developers, 246 tasks; AI allowed increased time by 19% while participants believed in a ~20% speedup — a warning that self-reported efficiency is not a valid outcome, hence WP29's logged time/cost.

# Stream 7 — Durable execution, failure recovery, observability, resource accounting (10)

Classic systems references: gray1986whystop (software and operations dominate outages; process pairs and transactions — verified only through a Semantic Scholar/DBLP conference record), dean2013tail (tail-tolerant techniques), helland2012idempotence (retries are safe if operations are idempotent), basiri2016chaos (IEEE Software 2016: chaos engineering principles), sigelman2010dapper (Google tech report: sampled, transparent tracing), burckhardt2021durablefunctions (OOPSLA 2021: formal semantics of durable workflows with persistent state and reliable progress). Agent-era references: dong2024agentops (taxonomy of traceable agent artifacts), khanal2026reliability (arXiv Mar. 2026: pass@1 and reliability diverge with task duration; 23,392 episodes), zhuang2026agentrewind (arXiv Aug. 2026: aligned checkpoints of context and environment for rewind/resume), erol2025costofpass (expected cost per correct solution; frontier cost-of-pass vs a human expert). No citable paper exists for Temporal/Cadence, Airflow or Argo; they are mentioned in WP29 as software, not as references.

# Stream 8 — Evidence-grounded generation and verification of scientific claims (11)

thorne2018fever (185,445 claims; Supported/Refuted/NEI; kappa 0.68) and wadden2020scifact (1.4K expert claims with rationales) define claim verification; bohnet2022attributedqa and gao2023rarr (ACL 2023) define attribution and post-hoc attribution/revision; min2023factscore (EMNLP 2023) gives atomic-fact precision (ChatGPT biographies 58% supported). wang2023selfconsistency (ICLR 2023) is in-model verification by agreement; huang2024selfcorrect (ICLR 2024) shows intrinsic self-correction fails without external feedback — the reason WP29's gates are external. Fabricated references: walters2023fabrication (55% of GPT-3.5 and 18% of GPT-4 citations fabricated in 84 reviews), alkaissi2023hallucinations (two case reports), sakai2026hallucitation (nearly 300 ACL/NAACL/EMNLP 2024-25 papers with at least one hallucinated citation; >100 accepted), zhao2026hallucinationswild (111 M references audited; ~146,932 hallucinated citations in 2025). Caveats: Walters & Wilder used 2023 models; Zhao et al. is a preprint with a conservative detection pipeline whose false-negative rate is unknown.

# Prior use of "protocol-guided autonomous research" and "PGAR"

Searched 2026-10-29/30 (web search, exact-phrase; arXiv listing titles via the API). Findings:

• The exact phrase "protocol-guided autonomous research" returned no paper, system or project using it. The nearest hits are (i) XScientist: A Git-Like Research Protocol for Long-Running Autonomous Scientific Discovery (arXiv:2607.12301, July 2026), which uses "protocol" for a provenance/hand-off standard for research state, and (ii) ProtoPilot (arXiv:2606.31763, June 2026), where "protocol" means a wet-lab procedure to be generated and executed by robots. Neither uses the phrase, and neither runs a controlled comparison of instruction structures.
• The acronym "PGAR" in the AI/science-agent context returned no match (results were generic surveys of autonomous research agents). The acronym therefore appears unused in this field as of the search date; it is used elsewhere for unrelated things (e.g., gene and company names), which WP29 should mention to avoid confusion.

# Closest prior work (honest qualification of novelty)

What is already established: (1) end-to-end autonomous pipelines can produce reviewable papers at low cost (Lu et al. 2024/2026; Yamada et al. 2025); (2) staging plus human feedback improves outcomes and reduces cost (Schmidgall et al. 2025); (3) traceable, human-verifiable evidence chains can be built into such pipelines (Ifargan et al. 2025 data-to-paper); (4) audits show these systems commit leakage, metric misuse, post-hoc selection and citation errors that are visible in traces but not in papers (Luo et al. 2025; Beel et al. 2025; Ding et al. 2026); (5) rigorous per-task evaluation with executable oracles, cost accounting and validity checklists exists (Chen et al. 2025; Starace et al. 2025; Kapoor et al. 2024/2025; Zhu et al. 2025); (6) in July 2026 XScientist proposed a git-like research protocol with gates and claim-evidence closure, i.e., the artifact side of what WP29 calls a protocol. What WP29 adds, and should claim no more than: a controlled, multi-arm comparison in which the executor (Claude Code, isolated, bare-mode) is held fixed and the instruction regime is varied — open instruction vs explicit protocol vs protocol + executable gates + persistent evidence vs the same plus separate review — with oracles protected from the executor and outcomes defined as scientific validity rather than reviewer score. No prior work found here randomizes protocol structure while holding the agent constant; the closest designs either vary the model/framework (benchmarks), vary human involvement (Agent Laboratory), or propose a protocol without an experiment (XScientist, data-to-paper). The novelty is the experimental design and its measurement, not the idea that protocols help.