1# WP29 — Source notes for the literature review23Scope: eight streams, 84 verified references (`literature_matrix.csv`, `references_verified.bib`), plus a novelty matrix of the closest systems (`novelty_matrix.csv`). Every reference was verified on 2026-10-30 (UTC) against a primary metadata source — Crossref (`api.crossref.org/works/<DOI>`), the arXiv API (`export.arxiv.org/api/query?id_list=`), Semantic Scholar (`api.semanticscholar.org/graph/v1/paper/...`) and, for ICLR/NeurIPS acceptance, OpenReview (`api2.openreview.net`) — and every DOI was checked to resolve at `doi.org` (all returned HTTP 302). Raw API responses are kept in `verify_cache/`; `build_literature_matrix.py` regenerates the CSV and the BibTeX from those cached records only. "Key claim" sentences are drawn from the abstracts retrieved through those APIs (or PubMed for three journal abstracts); where I did not read enough to summarize safely the matrix says `n/a` (one entry: the Nature 2026 version of The AI Scientist, whose abstract sits behind the publisher's login — the arXiv 2024 abstract is used for the system description instead).45Rule applied for `peer_reviewed`: "yes" only when a journal/proceedings record exists in Crossref, Semantic Scholar's `publicationVenue`, OpenReview, or the ICLR proceedings site; "no" for arXiv-only items; "unknown" for editorial/bulletin venues (Cortex editorial, American Scientist, CACM practice articles, ACM SIGIR Forum, Cureus editorial, the NASEM consensus report, and Gray 1986 whose only record is a Semantic Scholar/DBLP conference entry).67Dropped as unverifiable within the allowed sources (not cited): Oppenheimer, Ganapathi & Patterson (2003, USITS) and Yuan et al. (2014, OSDI) — no Crossref record and Semantic Scholar search was rate-limited (HTTP 429) on three attempts; Linardon et al. (2025, JMIR Mental Health, DOI 10.2196/80371) was verified but left out as redundant with Walters & Wilder; SWE-bench Verified is a vendor blog post with no citable metadata record and is discussed only through Zhu et al. (2025), who audit it. Two candidate IDs I had wrong were corrected during verification (ChemCrow DOI ends in `-00832-8`, not `-x`; DiscoveryBench is arXiv:2407.01725, not 2407.01214).89---1011## Stream 1 — Autonomous scientific discovery and agentic research systems (14)1213**lu2024aiscientist / lu2026aiscientist.** The AI Scientist (arXiv Aug. 2024; the line of work is now published in *Nature* as "Towards end-to-end automation of AI research", DOI 10.1038/s41586-026-10265-5) chains idea generation, code editing, experiment execution, plotting, paper writing and a simulated review; the preprint reports less than $15 per paper and an automated reviewer with near-human agreement on scores. Design caveats for WP29: a single pipeline with no control arm, human-written code templates, and an *in-system* reviewer (self-preference risk; see Panickssery et al.). Independent audits (Beel et al. 2025; Luo, Kasirzadeh & Shah 2025) found novelty misclassification, execution failures and leakage/metric misuse. I did not retrieve the Nature abstract, so no headline number is attributed to the 2026 version.1415**yamada2025aiscientistv2.** Removes the template dependence, uses a progressive agentic tree search managed by an experiment-manager agent and a VLM feedback loop for figures. Evidence of validity is one ICLR-2025-workshop submission (of three) scoring above the average acceptance threshold; n = 3 manuscripts, no uncertainty quantification, cost not given in the abstract. Not peer reviewed as a system paper.1617**schmidgall2025agentlab.** Agent Laboratory (Findings of EMNLP 2025) stages literature review, experimentation and report writing and lets humans give feedback at each stage. Findings: o1-preview gave the best outcomes, human feedback at each stage significantly improved quality, and expenses were 84% lower than prior autonomous methods. Units are human survey ratings and automated review scores; the number of raters is modest and the "quality" construct is reviewer score, not verified scientific validity. This is the closest evidence that *structure* (staging + feedback) matters, but the manipulated factor is human co-piloting, not the instruction protocol.1819**boiko2023coscientist.** Nature 2023. GPT-4 agent with web/docs search, code execution and lab-automation tools executes six chemistry tasks including successful Pd-catalysed cross-coupling optimisation. The oracle is physical; humans supervise execution. Establishes feasibility of tool-using autonomous experimentation, not a comparison of protocols.2021**bran2024chemcrow.** Nature Machine Intelligence 2024. 18 tools + GPT-4; syntheses of an insect repellent and three organocatalysts; evaluation by both LLM and expert assessment — an early acknowledgement that LLM-only evaluation is insufficient.2223**ifargan2025datatopaper.** NEJM AI 2025 (arXiv 2024). data-to-paper walks LLM agents from annotated data to manuscripts while *programmatically backtracing information flow* so humans can verify every number; autopilot vs copilot modes. Closest prior to WP29's "persistent evidence records", but it is a single pipeline (no arm without traceability) and verification is human, not an executable gate.2425**gottweis2026coscientist.** Nature 2026 (arXiv Feb. 2025). Gemini-based multi-agent generate/critique/rank/evolve loop with tournament evolution and test-time-compute scaling for hypothesis generation; critique is in-system, validation is downstream wet-lab work by collaborators. Scope is hypothesis generation, not executed computational research.2627**swanson2025virtuallab.** Nature 2025. LLM principal-investigator agent runs meetings with specialist agents and a critic; humans guide and run wet-lab validation of designed nanobodies. Shows a human-oversight design point; no controlled comparison.2829**mitchener2025kosmos.** arXiv Nov. 2025. Runs up to 12 h, ~200 rollouts, with a structured world model shared between analysis and literature agents. Important for long-horizon coherence; accuracy is judged by expert reading of reports; no executable validity gates.3031**luo2025pitfalls.** arXiv Sept. 2025 (NeurIPS 2025 AI-for-Science workshop spotlight per the conference site; counted as not peer reviewed). Defines four failure modes — inappropriate benchmark selection, data leakage, metric misuse, post-hoc selection bias — and finds several in two open-source AI scientist systems via controlled experiments. Directly motivates WP29's gates; caveat: two systems, authors' own rubric.3233**beel2025sakana.** ACM SIGIR Forum 2025 (bulletin; editorial review status unknown). Independent re-run of the AI Scientist: keyword-only literature search, ideas wrongly classified as novel, experiment failures. Small-n qualitative audit.3435**luo2026xscientist.** arXiv July 2026. Git-like protocol: typed content-addressed research objects, immutable checkpoints, claim-evidence closure, authority-aware gates; the authors stress a passing integrity check is not scientific truth. Conceptually the nearest neighbour to WP29's protocol + evidence records, but it is a protocol artifact/implementation, not an experiment.3637**ding2026verificationgap.** arXiv June 2026 survey: 125 screened, 35 included, 26 full-text coded on seven audit dimensions; code release is common but reproducibility-grade and claim-verification artifacts are rare. Provides the vocabulary ("verification gap") and audit dimensions WP29 reports against.3839## Stream 2 — Language agents for data analysis and scientific programming (9)4041**chen2025scienceagentbench** (ICLR 2025 poster; OpenReview record). 102 tasks from 44 papers, expert-validated, program-file outputs, execution-based metrics plus cost, two contamination mitigations. Best agent 32.4% (34.3% with expert knowledge); o1-preview 42.2% at >10x cost; three attempts per task. The explicit call to assess individual tasks "before making bold claims on end-to-end automation" is WP29's stance.4243**majumder2025discoverybench** (ICLR 2025). 264 real + 903 synthetic tasks; each task = dataset + metadata + discovery goal; hypothesis-level grading. I did not read the headline success rate, so none is quoted.4445**jing2025dsbench** (ICLR 2025). 466 analysis + 74 modeling tasks from Eloquence/Kaggle with long contexts; realism argument; headline rate not quoted.4647**huang2024mlagentbench** (ICML 2024). 13 experimentation tasks; ReAct agent; Claude 3 Opus best success rate. Small task count; success is metric improvement.4849**gu2024blade** (EMNLP 2024). 12 datasets/questions; ground truth from independent expert analyses; evaluates analysis *decisions* under multiple valid paths — the same oracle problem WP29's rubrics face.5051**jimenez2024swebench** (ICLR 2024). 2,294 GitHub issue/PR problems; test-based grading. Zhu et al. (2025) later show SWE-bench Verified's tests are insufficient — a caution for any test-as-oracle design.5253**chan2025mlebench** (ICLR 2025; proceedings page verified). 75 Kaggle competitions, human leaderboard baselines; best setup medals in 16.9%; resource-scaling and contamination analyses.5455**siegel2024corebench** (TMLR per Semantic Scholar). 270 reproduction tasks from 90 papers in three disciplines; links agent evaluation to computational reproducibility.5657**starace2025paperbench** (ICML 2025). 20 ICML papers, 8,316 rubric items co-developed with authors, LLM judge validated on a separate judge benchmark — the closest precedent for validated rubric oracles.5859## Stream 3 — Workflows, provenance, computational reproducibility (10)6061**koster2012snakemake** (Bioinformatics 2012) and **ditommaso2017nextflow** (Nature Biotechnology 2017, correspondence) are the canonical declarative/dataflow workflow engines; both argue that reproducibility comes from re-executable, environment-pinned workflow definitions. **missier2013prov** (EDBT 2013) summarises the W3C PROV family (PROV-DM/O/N/constraints) — the vocabulary (entity/activity/agent) that WP29's evidence records instantiate. **chirigati2016reprozip** (SIGMOD 2016 demo) captures execution provenance automatically into a re-executable package, i.e., evidence captured by the system rather than self-reported by the agent.6263**sandve2013tensimplerules** (PLoS Comput Biol 2013): rules include tracking how every result was produced, avoiding manual data manipulation, archiving program versions and seeds, and public access to scripts and results — WP29 turns these into machine-checked gates. **peng2011reproducible** (Science 2011) frames reproducibility as the minimum standard when replication is impossible. **nasem2019reproducibility** (NASEM 2019) fixes the definitions: reproducibility = same data/code/conditions, replicability = new data; WP29 uses these.6465**stodden2018journalpolicy** (PNAS 2018): random sample of 204 *Science* papers under a data/code-on-request policy; artifacts obtained for 44%, findings reproduced for 26% — policy without executable verification fails. **trisovic2022codequality** (Scientific Data 2022): >9,000 R files; 74% failed initially, 56% after cleaning. **krafczyk2021reproduction** (Phil Trans A 2021): three principles (transparency, re-executability, determinism), 12 guidelines, Reproduction Package. Caveats: Stodden's sample is one journal/one policy; Trisovic's failure definition is "any error on clean re-execution".6667## Stream 4 — Preregistration, registered reports, specification discipline (10)6869**nosek2018preregistration** (PNAS 2018): preregistration separates prediction from postdiction. **chambers2013registeredreports** (Cortex 2013 editorial) and **chambers2022registeredreports** (Nature Human Behaviour 2022 review): protocols reviewed and pre-accepted before data; early evidence they work; not a universal cure. **simmons2011falsepositive** (Psychological Science 2011): simulations and two experiments show researcher degrees of freedom make false positives easy; six author requirements + four reviewer guidelines. **gelman2014statisticalcrisis** (American Scientist 2014): garden of forking paths — data-dependent choices inflate false positives without intent. **kerr1998harking**: presenting post hoc hypotheses as a priori. **munafo2017manifesto** and **nosek2015top**: the measures and TOP transparency tiers. **scheel2021excess** (AMPPS 2021): 96% positive first hypotheses in 152 standard papers vs 44% in 71 Registered Reports — the clearest quantitative effect of specification-before-results, with the caveat that RR authors self-select. **osc2015reproducibility** (Science 2015): 100 replications, roughly one-third to one-half of findings replicated depending on criterion.7071Relevance: WP29's "protocol" arm is, functionally, a preregistration imposed on the agent; the "open instruction" arm is the garden of forking paths with an agent walking it.7273## Stream 5 — Agent evaluation, contamination, evaluator validity (10)7475**liu2024agentbench** (ICLR 2024): eight environments; instruction-following and long-horizon reasoning are the main failure causes. **yao2024taubench** (arXiv 2024; venue unverified): state-based outcome checking and pass^k; gpt-4o <50% success, pass^8 <25% retail — reliability over repeated runs is a separate quantity from success rate. **kapoor2024agentsthatmatter** (TMLR per Semantic Scholar): cost must be reported alongside accuracy; holdouts are often inadequate; model- vs downstream-developer needs conflated. **kapoor2025hal** (arXiv Oct. 2025): standardized harness, 21,730 rollouts, ~$40,000. **zhu2025abc** (arXiv July 2025): task/outcome-validity flaws (SWE-bench Verified insufficient tests; TAU-bench counts empty responses as success) mis-estimate performance by up to 100% relative; checklist cut overestimation on CVE-Bench by 33%. **sainz2023contamination** (Findings EMNLP 2023): contamination levels and the call to measure per benchmark.7677**zheng2023mtbench** (NeurIPS 2023 D&B): LLM judges have position, verbosity and self-enhancement biases; GPT-4 reaches >80% agreement with humans, equal to human-human. **thakur2025judgingjudges** (GEM2 workshop 2025): even the best judges lag inter-human agreement; scores can differ by up to 5 points. **panickssery2024selfpreference** (NeurIPS 2024): self-recognition correlates linearly with self-preference. **li2025llmreval** (arXiv Oct. 2025): simulated research-and-review loop with human annotation shows pronounced misalignment between LLM reviewers and humans. Together these justify WP29's protected, executor-separated oracles and its refusal to let the executor's own model grade itself.7879## Stream 6 — Human-AI delegation, automation bias, intervention measurement (10)8081**parasuraman1997automation** (use/misuse/disuse/abuse) and **parasuraman2000levels** (four function classes, continuum of levels) give the vocabulary for placing WP29's arms. **goddard2012automationbias** (JAMIA 2012): systematic review of 74 studies; bias mediated by cognitive style, DSS design and experience. **bainbridge1983ironies**: automation leaves humans the hardest residual tasks. **bansal2021whole** (CHI 2021): explanations raise acceptance regardless of correctness. **bucinca2021trust** (CSCW 2021, N = 199): cognitive forcing reduces overreliance — the design principle behind requiring reviewers to execute checks rather than read agent summaries. **lubars2019delegability** (NeurIPS 2019): strong preference for machine-in-the-loop over full AI control; trust dominates. **lai2023humanai** (FAccT 2023): design space of empirical human-AI decision studies. **vaccaro2024combinations** (Nature Human Behaviour 2024): 106 experiments, 370 effect sizes; combinations on average below the best of human or AI alone, with gains in creation tasks. **becker2025metr** (arXiv July 2025): RCT, 16 developers, 246 tasks; AI allowed *increased* time by 19% while participants believed in a ~20% speedup — a warning that self-reported efficiency is not a valid outcome, hence WP29's logged time/cost.8283## Stream 7 — Durable execution, failure recovery, observability, resource accounting (10)8485Classic systems references: **gray1986whystop** (software and operations dominate outages; process pairs and transactions — verified only through a Semantic Scholar/DBLP conference record), **dean2013tail** (tail-tolerant techniques), **helland2012idempotence** (retries are safe if operations are idempotent), **basiri2016chaos** (IEEE Software 2016: chaos engineering principles), **sigelman2010dapper** (Google tech report: sampled, transparent tracing), **burckhardt2021durablefunctions** (OOPSLA 2021: formal semantics of durable workflows with persistent state and reliable progress). Agent-era references: **dong2024agentops** (taxonomy of traceable agent artifacts), **khanal2026reliability** (arXiv Mar. 2026: pass@1 and reliability diverge with task duration; 23,392 episodes), **zhuang2026agentrewind** (arXiv Aug. 2026: aligned checkpoints of context and environment for rewind/resume), **erol2025costofpass** (expected cost per correct solution; frontier cost-of-pass vs a human expert). No citable paper exists for Temporal/Cadence, Airflow or Argo; they are mentioned in WP29 as software, not as references.8687## Stream 8 — Evidence-grounded generation and verification of scientific claims (11)8889**thorne2018fever** (185,445 claims; Supported/Refuted/NEI; kappa 0.68) and **wadden2020scifact** (1.4K expert claims with rationales) define claim verification; **bohnet2022attributedqa** and **gao2023rarr** (ACL 2023) define attribution and post-hoc attribution/revision; **min2023factscore** (EMNLP 2023) gives atomic-fact precision (ChatGPT biographies 58% supported). **wang2023selfconsistency** (ICLR 2023) is in-model verification by agreement; **huang2024selfcorrect** (ICLR 2024) shows intrinsic self-correction fails without external feedback — the reason WP29's gates are external. Fabricated references: **walters2023fabrication** (55% of GPT-3.5 and 18% of GPT-4 citations fabricated in 84 reviews), **alkaissi2023hallucinations** (two case reports), **sakai2026hallucitation** (nearly 300 ACL/NAACL/EMNLP 2024-25 papers with at least one hallucinated citation; >100 accepted), **zhao2026hallucinationswild** (111 M references audited; ~146,932 hallucinated citations in 2025). Caveats: Walters & Wilder used 2023 models; Zhao et al. is a preprint with a conservative detection pipeline whose false-negative rate is unknown.9091---9293## Prior use of "protocol-guided autonomous research" and "PGAR"9495Searched 2026-10-29/30 (web search, exact-phrase; arXiv listing titles via the API). Findings:9697- The exact phrase **"protocol-guided autonomous research"** returned no paper, system or project using it. The nearest hits are (i) *XScientist: A Git-Like Research Protocol for Long-Running Autonomous Scientific Discovery* (arXiv:2607.12301, July 2026), which uses "protocol" for a provenance/hand-off standard for research state, and (ii) *ProtoPilot* (arXiv:2606.31763, June 2026), where "protocol" means a wet-lab procedure to be generated and executed by robots. Neither uses the phrase, and neither runs a controlled comparison of instruction structures.98- The acronym **"PGAR"** in the AI/science-agent context returned no match (results were generic surveys of autonomous research agents). The acronym therefore appears unused in this field as of the search date; it is used elsewhere for unrelated things (e.g., gene and company names), which WP29 should mention to avoid confusion.99100## Closest prior work (honest qualification of novelty)101102What is already established: (1) end-to-end autonomous pipelines can produce reviewable papers at low cost (Lu et al. 2024/2026; Yamada et al. 2025); (2) staging plus human feedback improves outcomes and reduces cost (Schmidgall et al. 2025); (3) traceable, human-verifiable evidence chains can be built into such pipelines (Ifargan et al. 2025 data-to-paper); (4) audits show these systems commit leakage, metric misuse, post-hoc selection and citation errors that are visible in traces but not in papers (Luo et al. 2025; Beel et al. 2025; Ding et al. 2026); (5) rigorous per-task evaluation with executable oracles, cost accounting and validity checklists exists (Chen et al. 2025; Starace et al. 2025; Kapoor et al. 2024/2025; Zhu et al. 2025); (6) in July 2026 XScientist proposed a git-like research protocol with gates and claim-evidence closure, i.e., the *artifact* side of what WP29 calls a protocol. What WP29 adds, and should claim no more than: a **controlled, multi-arm comparison** in which the executor (Claude Code, isolated, bare-mode) is held fixed and the instruction regime is varied — open instruction vs explicit protocol vs protocol + executable gates + persistent evidence vs the same plus separate review — with **oracles protected from the executor** and outcomes defined as scientific validity rather than reviewer score. No prior work found here randomizes protocol structure while holding the agent constant; the closest designs either vary the model/framework (benchmarks), vary human involvement (Agent Laboratory), or propose a protocol without an experiment (XScientist, data-to-paper). The novelty is the experimental design and its measurement, not the idea that protocols help.103