1ref_key,stream,authors_short,year,title,venue,venue_type,doi_or_arxiv,verification_source,verified_utc,relevance_to_wp29,key_claim_or_finding,peer_reviewed2lu2024aiscientist,1,Lu et al.,2024,The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery,arXiv preprint,preprint,arXiv:2408.06292,https://export.arxiv.org/api/query?id_list=2408.06292,2026-10-30T03:12:38Z,Reference end-to-end autonomous research system (idea to paper plus simulated review) against which WP29's protocol-guided design is positioned.,"Introduces The AI Scientist, which generates research ideas, writes code, executes experiments, visualizes results, writes a full paper and runs a simulated review, with each paper developed at a cost of less than $15 and an automated reviewer reported to achieve near-human performance on paper scores.",no3lu2026aiscientist,1,Lu et al.,2026,Towards end-to-end automation of AI research,Nature,journal,doi:10.1038/s41586-026-10265-5,https://api.crossref.org/works/10.1038/s41586-026-10265-5 ; doi.org HEAD None,2026-10-30T03:18:58Z,"Peer-reviewed (Nature, 2026) publication of The AI Scientist line of work, establishing that fully automated ML research pipelines are now part of the refereed literature.",n/a (Nature abstract not retrieved behind the publisher login; see lu2024aiscientist for the system description).,yes4yamada2025aiscientistv2,1,Yamada et al.,2025,The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search,arXiv preprint,preprint,arXiv:2504.08066,https://export.arxiv.org/api/query?id_list=2504.08066,2026-10-30T03:12:39Z,Closest fully autonomous end-to-end system; its tree-search workflow and in-system AI reviewer are the counterpoint to WP29's executable gates and separated review.,"Describes an end-to-end agentic system with progressive agentic tree search managed by an experiment-manager agent and a VLM feedback loop for figures; of three fully autonomous manuscripts submitted to an ICLR workshop, one scored above the average human acceptance threshold.",no5schmidgall2025agentlab,1,Schmidgall et al.,2025,Agent Laboratory: Using LLM Agents as Research Assistants,Findings of the Association for Computational Linguistics: EMNLP 2025,conference,doi:10.18653/v1/2025.findings-emnlp.320,https://api.crossref.org/works/10.18653/v1/2025.findings-emnlp.320 ; doi.org HEAD None ; https://export.arxiv.org/api/query?id_list=2501.04227,2026-10-30T03:18:58Z,"Staged (literature review, experimentation, report writing) autonomous research framework with optional human feedback; provides the human-in-the-loop and cost comparisons WP29 builds on.","Agent Laboratory takes a human-provided idea through literature review, experimentation and report writing; the authors report that o1-preview gives the best outcomes, that human feedback at each stage significantly improves research quality, and an 84% decrease in research expenses relative to previous autonomous research methods.",yes6boiko2023coscientist,1,Boiko et al.,2023,Autonomous chemical research with large language models,Nature,journal,doi:10.1038/s41586-023-06792-0,https://api.crossref.org/works/10.1038/s41586-023-06792-0 ; doi.org HEAD 302,2026-10-30T03:12:40Z,Landmark peer-reviewed demonstration of an LLM agent planning and executing laboratory experiments; defines the 'autonomous research' baseline WP29 compares to.,"Coscientist, driven by GPT-4 with internet/documentation search, code execution and experimental automation tools, autonomously designs, plans and performs experiments across six tasks, including successful reaction optimization of palladium-catalysed cross-couplings.",yes7bran2024chemcrow,1,M. Bran et al.,2024,Augmenting large language models with chemistry tools,Nature Machine Intelligence,journal,doi:10.1038/s42256-024-00832-8,https://api.crossref.org/works/10.1038/s42256-024-00832-8 ; doi.org HEAD 302,2026-10-30T03:15:06Z,"Tool-augmented LLM research agent whose evaluation combined LLM and expert assessment, an early instance of the evaluator-validity question WP29 addresses with protected oracles.","ChemCrow integrates 18 expert-designed tools with GPT-4, autonomously planned and executed syntheses of an insect repellent and three organocatalysts, and was evaluated with both LLM and expert assessments.",yes8ifargan2025datatopaper,1,Ifargan et al.,2025,Autonomous LLM-Driven Research — from Data to Human-Verifiable Research Papers,NEJM AI,journal,doi:10.1056/AIoa2400555,https://api.crossref.org/works/10.1056/AIoa2400555 ; doi.org HEAD 302 ; https://export.arxiv.org/api/query?id_list=2404.17605,2026-10-30T03:13:47Z,"Closest prior work on traceability: an autonomous data-to-paper platform that programmatically backtraces information flow for human verification, akin to WP29's persistent evidence records.","data-to-paper guides interacting LLM agents through a stepwise research process from annotated data to papers while programmatically backtracing information flow and allowing human oversight, and can run fully autonomously ('autopilot') or with human intervention.",yes9gottweis2026coscientist,1,Gottweis et al.,2026,Accelerating scientific discovery with Co-Scientist,Nature,journal,doi:10.1038/s41586-026-10644-y,https://api.crossref.org/works/10.1038/s41586-026-10644-y ; doi.org HEAD 302 ; https://export.arxiv.org/api/query?id_list=2502.18864,2026-10-30T03:13:46Z,Multi-agent hypothesis-generation system with generate/critique/refine loops; illustrates in-system self-critique rather than independent verification.,"Co-Scientist is a Gemini-based multi-agent system that continuously generates, critiques and refines research hypotheses using an asynchronous task framework and a tournament evolution process, scaled by test-time compute, to propose demonstrably novel hypotheses for experimental verification.",yes10swanson2025virtuallab,1,Swanson et al.,2025,The Virtual Lab of AI agents designs new SARS-CoV-2 nanobodies,Nature,journal,doi:10.1038/s41586-025-09442-9,https://api.crossref.org/works/10.1038/s41586-025-09442-9 ; doi.org HEAD 302,2026-10-30T03:12:48Z,AI-human research collaboration with an LLM principal-investigator agent and human oversight; a human-involvement design point for the novelty matrix.,"The Virtual Lab is an AI-human research collaboration in which an LLM Principal Investigator agent guides a team of LLM scientist agents to perform interdisciplinary research, applied to designing SARS-CoV-2 nanobodies.",yes11mitchener2025kosmos,1,Mitchener et al.,2025,Kosmos: An AI Scientist for Autonomous Discovery,arXiv preprint,preprint,arXiv:2511.02824,https://export.arxiv.org/api/query?id_list=2511.02824,2026-10-30T03:12:57Z,"Recent long-horizon AI scientist (up to 12 h, 200 rollouts); documents the coherence/verification challenges of long autonomous runs that WP29's gates target.","Kosmos runs for up to 12 hours performing cycles of parallel data analysis, literature search and hypothesis generation, using a structured world model shared between agents to pursue an objective coherently over 200 agent rollouts before synthesizing reports.",no12luo2025pitfalls,1,Luo et al.,2025,"The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systems",arXiv preprint,preprint,arXiv:2509.08713,https://export.arxiv.org/api/query?id_list=2509.08713,2026-10-30T03:16:07Z,"Directly motivates WP29: shows AI scientist systems commit benchmark selection, leakage, metric-misuse and post-hoc selection errors that only traces reveal.","Identifies four failure modes in AI scientist systems (inappropriate benchmark selection, data leakage, metric misuse and post-hoc selection bias) and, via controlled experiments on two open-source systems, finds several of them present.",no13beel2025sakana,1,Beel et al.,2025,"Evaluating Sakana's AI Scientist: Bold Claims, Mixed Results, and a Promising Future?",ACM SIGIR Forum,journal,doi:10.1145/3769733.3769747,https://api.crossref.org/works/10.1145/3769733.3769747 ; doi.org HEAD None ; https://export.arxiv.org/api/query?id_list=2502.14297,2026-10-30T03:18:59Z,Independent third-party evaluation of The AI Scientist; evidence that unverified autonomous outputs contain novelty and execution errors.,"An independent evaluation of the AI Scientist found several critical shortcomings, including a literature review based on simplistic keyword searches that led to poor novelty assessments, with several generated ideas incorrectly classified as novel.",unknown14luo2026xscientist,1,Luo,2026,XScientist: A Git-Like Research Protocol for Long-Running Autonomous Scientific Discovery,arXiv preprint,preprint,arXiv:2607.12301,https://export.arxiv.org/api/query?id_list=2607.12301,2026-10-30T03:13:42Z,"Closest 2026 design to a research protocol: git-like typed research state with checkpoints, claim-evidence closure and gates; differs from WP29 in being a protocol artifact rather than a controlled comparison of protocol arms.","Proposes a local-first, git-like protocol that treats research state rather than a manuscript as the unit of continuation, representing hypotheses, attempts, claims, reviews and handoffs as typed content-addressed objects with immutable checkpoints, claim-evidence closure and authority-aware gates, without treating a passing integrity check as scientific truth.",no15ding2026verificationgap,1,Ding et al.,2026,Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap,arXiv preprint,preprint,arXiv:2608.05179,https://export.arxiv.org/api/query?id_list=2608.05179,2026-10-30T03:13:43Z,"Survey that names the 'verification gap' in AI scientist systems; supplies the audit dimensions (evaluation method, artifacts, human-in-the-loop points, result-selection disclosure) WP29 reports.","Screens 125 works and codes 26 (24 runnable systems) on seven audit dimensions, finding that code release is now common but reproducibility-grade and claim-verification artifacts remain rare.",no16chen2025scienceagentbench,2,Chen et al.,2024,ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery,ICLR 2025,conference,arXiv:2410.05080,"https://export.arxiv.org/api/query?id_list=2410.05080 ; https://api.semanticscholar.org/graph/v1/paper/arXiv:2410.05080 (venue='arXiv.org', type=None) ; https://api2.openreview.net/notes/search?term=ScienceAgentBench (ICLR 2025 Poster record)",2026-10-30T03:13:01Z,"Benchmark of expert-validated data-driven discovery tasks with program-level outputs, execution metrics and cost; the task style and contamination mitigations inform WP29's task bank.","Presents 102 tasks extracted from 44 publications in four disciplines and validated by nine experts; the best agent solves 32.4% of tasks independently (34.3% with expert knowledge), o1-preview reaches 42.2% at more than 10 times the cost, and two data-contamination mitigation strategies are proposed.",yes17majumder2025discoverybench,2,Majumder et al.,2024,DiscoveryBench: Towards Data-Driven Discovery with Large Language Models,ICLR 2025,conference,arXiv:2407.01725,"https://export.arxiv.org/api/query?id_list=2407.01725 ; https://api.semanticscholar.org/graph/v1/paper/arXiv:2407.01725 (venue='International Conference on Learning Representations', type='conference') ; https://api2.openreview.net/notes/search?term=DiscoveryBench (ICLR 2025 Poster record)",2026-10-30T03:15:06Z,"Formalizes multi-step data-driven discovery as dataset + metadata + goal, the same unit of analysis as WP29's research tasks.","DiscoveryBench formalizes the multi-step process of data-driven discovery with 264 tasks across six domains derived from published papers plus 903 synthetic tasks, each defined by a dataset, its metadata and a natural-language discovery goal.",yes18jing2025dsbench,2,Jing et al.,2024,DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?,ICLR 2025,conference,arXiv:2409.07703,"https://export.arxiv.org/api/query?id_list=2409.07703 ; https://api.semanticscholar.org/graph/v1/paper/arXiv:2409.07703 (venue='', type=None) ; https://api2.openreview.net/notes/search?term=DSBench (ICLR 2025 Poster record)",2026-10-30T03:13:02Z,"Realistic data-science agent benchmark (long contexts, large data) used to characterize executor capability independent of protocol.","DSBench comprises 466 data-analysis and 74 data-modeling tasks sourced from Eloquence and Kaggle competitions, with long contexts, multimodal task backgrounds and reasoning over large data files.",yes19huang2024mlagentbench,2,Huang et al.,2023,MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation,ICML 2024,conference,arXiv:2310.03302,"https://export.arxiv.org/api/query?id_list=2310.03302 ; https://api.semanticscholar.org/graph/v1/paper/arXiv:2310.03302 (venue='International Conference on Machine Learning', type='conference')",2026-10-30T03:13:03Z,Evaluates agents on ML experimentation loops with file/code actions; a precursor of WP29's executor arms.,"MLAgentBench is a suite of 13 ML experimentation tasks (from CIFAR-10 improvement to BabyLM) in which a ReAct-based agent can read/write files, execute code and inspect outputs; a Claude 3 Opus agent had the best success rate.",yes20gu2024blade,2,Gu et al.,2024,BLADE: Benchmarking Language Model Agents for Data-Driven Science,EMNLP 2024,conference,arXiv:2408.09667,"https://export.arxiv.org/api/query?id_list=2408.09667 ; https://api.semanticscholar.org/graph/v1/paper/arXiv:2408.09667 (venue='Conference on Empirical Methods in Natural Language Processing', type='conference')",2026-10-30T03:13:05Z,"Evaluates analysis decisions (variables, transformations, models) against independent expert analyses, the multi-path ground truth WP29's oracles must handle.","BLADE evaluates agents' multifaceted approaches to open-ended research questions using 12 datasets and questions from the literature, with ground truth collected from independent analyses by expert data scientists.",yes21jimenez2024swebench,2,Jimenez et al.,2023,SWE-bench: Can Language Models Resolve Real-World GitHub Issues?,ICLR 2024,conference,arXiv:2310.06770,"https://export.arxiv.org/api/query?id_list=2310.06770 ; https://api.semanticscholar.org/graph/v1/paper/arXiv:2310.06770 (venue='International Conference on Learning Representations', type='conference')",2026-10-30T03:13:08Z,Canonical execution-verified coding benchmark; its test-based grading is the model for WP29's executable verification gates.,"SWE-bench consists of 2,294 software-engineering problems drawn from real GitHub issues and pull requests across 12 Python repositories, where resolving an issue frequently requires coordinating changes across multiple functions, classes and files.",yes22chan2025mlebench,2,Chan et al.,2024,MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering,ICLR 2025,conference,arXiv:2410.07095,"https://export.arxiv.org/api/query?id_list=2410.07095 ; https://api.semanticscholar.org/graph/v1/paper/arXiv:2410.07095 (venue='arXiv.org', type=None) ; https://proceedings.iclr.cc/paper_files/paper/2025/hash/7e3767db483c942b883eb4f8cfb74e31-Abstract-Conference.html",2026-10-30T03:13:09Z,"Agentic ML-engineering benchmark with human baselines, resource-scaling and contamination analyses that WP29's resource accounting parallels.","MLE-bench curates 75 Kaggle competitions with human leaderboard baselines; the best setup (o1-preview with AIDE scaffolding) reaches at least bronze-medal level in 16.9% of competitions, and the paper also studies resource scaling and pre-training contamination.",yes23siegel2024corebench,2,Siegel et al.,2024,CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark,Transactions on Machine Learning Research,journal,arXiv:2409.11363,https://export.arxiv.org/api/query?id_list=2409.11363 ; https://api.semanticscholar.org/graph/v1/paper/arXiv:2409.11363 (venue='Trans. Mach. Learn. Res.'),2026-10-30T03:13:10Z,"Benchmarks agents on computational reproducibility of published papers, linking agent evaluation to the reproducibility stream.","CORE-Bench contains 270 tasks based on 90 scientific papers across computer science, social science and medicine, measuring agents' accuracy at reproducing published results from provided code and data at three difficulty levels.",yes24starace2025paperbench,2,Starace et al.,2025,PaperBench: Evaluating AI's Ability to Replicate AI Research,ICML 2025,conference,arXiv:2504.01848,"https://export.arxiv.org/api/query?id_list=2504.01848 ; https://api.semanticscholar.org/graph/v1/paper/arXiv:2504.01848 (venue='International Conference on Machine Learning', type='conference')",2026-10-30T03:13:11Z,Rubric-decomposed replication grading with a validated LLM judge; closest precedent for WP29's rubric oracles and judge validation.,"PaperBench asks agents to replicate 20 ICML 2024 papers from scratch, graded against 8,316 rubric items co-developed with the papers' authors, with an LLM-based judge whose performance is assessed on a separate judge benchmark.",yes25koster2012snakemake,3,Köster & Rahmann,2012,Snakemake—a scalable bioinformatics workflow engine,Bioinformatics,journal,doi:10.1093/bioinformatics/bts480,https://api.crossref.org/works/10.1093/bioinformatics/bts480 ; doi.org HEAD 302,2026-10-30T03:13:47Z,"Rule-based workflow engine; WP29's protocol stages are an analogue of declarative, re-executable workflow rules.",Snakemake provides a readable Python-based workflow definition language and an execution environment that scales from single-core workstations to compute clusters without modifying the workflow.,yes26ditommaso2017nextflow,3,Di Tommaso et al.,2017,Nextflow enables reproducible computational workflows,Nature Biotechnology,journal,doi:10.1038/nbt.3820,https://api.crossref.org/works/10.1038/nbt.3820 ; doi.org HEAD 302,2026-10-30T03:13:48Z,Container-based dataflow workflow manager for reproducible pipelines; reference for execution isolation in WP29's executors.,Argues that combining dataflow programming with software containers (Nextflow) yields computational workflows that are reproducible across platforms.,yes27missier2013prov,3,Missier et al.,2013,The W3C PROV family of specifications for modelling provenance metadata,Proceedings of the 16th International Conference on Extending Database Technology,conference,doi:10.1145/2452376.2452478,https://api.crossref.org/works/10.1145/2452376.2452478 ; doi.org HEAD 302,2026-10-30T03:13:49Z,"W3C PROV family defines the provenance vocabulary (entities, activities, agents) that WP29's evidence records instantiate.","Overview of the W3C PROV family of specifications (data model, ontology, notation and constraints) for interoperable representation and exchange of provenance metadata.",yes28chirigati2016reprozip,3,Chirigati et al.,2016,ReproZip,Proceedings of the 2016 International Conference on Management of Data,conference,doi:10.1145/2882903.2899401,https://api.crossref.org/works/10.1145/2882903.2899401 ; doi.org HEAD 302,2026-10-30T03:13:49Z,Automatic capture of execution provenance into re-executable packages; a model for evidence capture without relying on the agent's self-report.,"ReproZip automatically captures the provenance of a command-line experiment (files, libraries, environment) and packages it so that it can be re-executed elsewhere.",yes29sandve2013tensimplerules,3,Sandve et al.,2013,Ten Simple Rules for Reproducible Computational Research,PLoS Computational Biology,journal,doi:10.1371/journal.pcbi.1003285,https://api.crossref.org/works/10.1371/journal.pcbi.1003285 ; doi.org HEAD 302,2026-10-30T03:13:50Z,"Widely cited rules (track provenance, avoid manual steps, record versions and seeds) that WP29 operationalizes as machine-checked gates.","Proposes ten rules for reproducible computational research, including tracking how every result was produced, avoiding manual data manipulation, archiving exact program versions and random seeds, and providing public access to scripts, runs and results.",yes30stodden2018journalpolicy,3,Stodden et al.,2018,An empirical analysis of journal policy effectiveness for computational reproducibility,Proceedings of the National Academy of Sciences,journal,doi:10.1073/pnas.1708290115,https://api.crossref.org/works/10.1073/pnas.1708290115 ; doi.org HEAD 302,2026-10-30T03:13:50Z,Empirical reproducibility failure rates under policy-only enforcement; motivates executable rather than declarative verification.,"In a random sample of 204 Science papers subject to a data/code-on-request policy, the authors obtained artifacts from 44% of papers and could reproduce the findings for 26%.",yes31peng2011reproducible,3,Peng,2011,Reproducible Research in Computational Science,Science,journal,doi:10.1126/science.1213847,https://api.crossref.org/works/10.1126/science.1213847 ; doi.org HEAD 302,2026-10-30T03:13:51Z,"Defines reproducibility (data + code available) as the minimum standard for computational claims, the standard WP29's evidence records target.",Reproducibility has the potential to serve as a minimum standard for judging scientific claims when full independent replication of a study is not possible.,yes32nasem2019reproducibility,3,"National Academies of Sciences, Engineering, and Medicine",2019,Reproducibility and Replicability in Science,National Academies Press,report,doi:10.17226/25303,https://api.crossref.org/works/10.17226/25303 ; doi.org HEAD 302,2026-10-30T03:13:51Z,Authoritative definitions of reproducibility vs replicability used in WP29's outcome definitions.,"Defines reproducibility as obtaining consistent computational results using the same input data, computational steps, methods, code and conditions of analysis, and replicability as obtaining consistent results across studies aimed at the same question with new data.",unknown33krafczyk2021reproduction,3,Krafczyk et al.,2021,Learning from reproducing computational results: introducing three principles and the <i>Reproduction Package</i>,"Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences",journal,doi:10.1098/rsta.2020.0069,https://api.crossref.org/works/10.1098/rsta.2020.0069 ; doi.org HEAD 302,2026-10-30T03:13:52Z,"Principles (transparency, re-executability, determinism) and a Reproduction Package formalism close to WP29's release bundle.","From reproducing seven articles the authors derive three principles (transparency about how results are produced, ease of re-executability, determinism), 12 guidelines, and a Reproduction Package formalism for sharing artifacts.",yes34trisovic2022codequality,3,Trisovic et al.,2022,A large-scale study on research code quality and execution,Scientific Data,journal,doi:10.1038/s41597-022-01143-6,https://api.crossref.org/works/10.1038/s41597-022-01143-6 ; doi.org HEAD 302,2026-10-30T03:13:53Z,"Large-scale evidence that deposited research code usually fails to run, justifying execution-based verification gates.","Executing over 9,000 R files from more than 2,000 Harvard Dataverse replication datasets, 74% failed to complete without error initially and 56% still failed after automatic code cleaning.",yes35nosek2018preregistration,4,Nosek et al.,2018,The preregistration revolution,Proceedings of the National Academy of Sciences,journal,doi:10.1073/pnas.1708274114,https://api.crossref.org/works/10.1073/pnas.1708274114 ; doi.org HEAD 302,2026-10-30T03:13:55Z,Preregistration as prediction/postdiction separation is the direct analogue of WP29's protocol fixed before execution.,"Defining research questions and the analysis plan before observing outcomes (preregistration) distinguishes analyses that result from predictions from those that result from postdictions, and practical strategies exist for cases that fall short of the ideal.",yes36chambers2013registeredreports,4,Chambers,2013,Registered Reports: A new publishing initiative at Cortex,Cortex,journal,doi:10.1016/j.cortex.2012.12.016,https://api.crossref.org/works/10.1016/j.cortex.2012.12.016 ; doi.org HEAD 302,2026-10-30T03:13:56Z,"Origin of Registered Reports: review of the protocol before results, the publishing analogue of WP29's pre-execution gate.","Introduces Registered Reports at Cortex, in which study protocols are peer reviewed and provisionally accepted before data collection.",unknown37chambers2022registeredreports,4,Chambers & Tzavella,2021,"The past, present and future of Registered Reports",Nature Human Behaviour,journal,doi:10.1038/s41562-021-01193-7,https://api.crossref.org/works/10.1038/s41562-021-01193-7 ; doi.org HEAD 302,2026-10-30T03:13:57Z,Reviews evidence that pre-acceptance of protocols reduces reporting bias; supports the 'specification before execution' arm.,Registered Reports are a publication format in which study proposals are peer reviewed and pre-accepted before research is undertaken; early evidence indicates they work as intended although they are not a universal solution for irreproducibility.,yes38simmons2011falsepositive,4,Simmons et al.,2011,False-Positive Psychology,Psychological Science,journal,doi:10.1177/0956797611417632,https://api.crossref.org/works/10.1177/0956797611417632 ; doi.org HEAD 302,2026-10-30T03:13:57Z,Researcher degrees of freedom inflate false positives; WP29's locked analysis specification removes these from the agent.,"Flexibility in data collection, analysis and reporting dramatically increases actual false-positive rates, so that a researcher is often more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not; six disclosure requirements for authors and four reviewer guidelines are proposed.",yes39munafo2017manifesto,4,Munafò et al.,2017,A manifesto for reproducible science,Nature Human Behaviour,journal,doi:10.1038/s41562-016-0021,https://api.crossref.org/works/10.1038/s41562-016-0021 ; doi.org HEAD 302,2026-10-30T03:13:57Z,"Umbrella statement of reproducibility measures (methods, reporting, evaluation, incentives) that WP29 adapts to autonomous agents.","Argues for measures optimizing methods, reporting and dissemination, reproducibility, evaluation and incentives, noting evidence from simulations and empirical studies but the need for iterative evaluation of their adoption.",yes40gelman2014statisticalcrisis,4,Gelman & Loken,2014,The Statistical Crisis in Science,American Scientist,journal,doi:10.1511/2014.111.460,https://api.crossref.org/works/10.1511/2014.111.460 ; doi.org HEAD 302,2026-10-30T03:13:58Z,"Garden of forking paths: data-dependent analysis choices invalidate inference even without intent, which an agent in an open instruction arm is free to make.",Argues that data-dependent choices of analysis (the 'garden of forking paths') can produce spurious statistically significant findings even when researchers do not deliberately fish for significance.,unknown41nosek2015top,4,Nosek et al.,2015,Promoting an open research culture,Science,journal,doi:10.1126/science.aab2374,https://api.crossref.org/works/10.1126/science.aab2374 ; doi.org HEAD 302,2026-10-30T03:13:59Z,"TOP guidelines define graded transparency standards (preregistration, data, code) that WP29's protocol tiers mirror.","Proposes journal author guidelines (the TOP guidelines) to promote transparency, openness and reproducibility, noting that although scientists embrace these norms they are not routine in practice.",yes42scheel2021excess,4,Scheel et al.,2021,An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports,Advances in Methods and Practices in Psychological Science,journal,doi:10.1177/25152459211007467,https://api.crossref.org/works/10.1177/25152459211007467 ; doi.org HEAD 302,2026-10-30T03:14:00Z,Quantifies how pre-specified protocols change the result distribution; benchmark for interpreting WP29's arm differences.,"Comparing 71 published Registered Reports with 152 standard hypothesis-testing papers, the first hypothesis was positive in 96% of standard reports but only 44% of Registered Reports.",yes43osc2015reproducibility,4,Open Science Collaboration,2015,Estimating the reproducibility of psychological science,Science,journal,doi:10.1126/science.aac4716,https://api.crossref.org/works/10.1126/science.aac4716 ; doi.org HEAD 302,2026-10-30T03:14:00Z,Baseline replication rates in a human literature; provides a human reference for WP29's validity outcomes.,"Replications of 100 experiments from 2008 papers in three psychology journals found that about one-third to one-half of original findings were also observed in the replication, depending on the criterion.",yes44kerr1998harking,4,Kerr,1998,HARKing: Hypothesizing After the Results are Known,Personality and Social Psychology Review,journal,doi:10.1207/s15327957pspr0203_4,https://api.crossref.org/works/10.1207/s15327957pspr0203_4 ; doi.org HEAD 302,2026-10-30T03:14:01Z,HARKing (presenting post hoc hypotheses as a priori) is a failure mode an unconstrained agent can reproduce when writing up.,"Defines HARKing (hypothesizing after the results are known) as presenting post hoc hypotheses as if they were a priori, and discusses its prevalence and costs.",yes45liu2024agentbench,5,Liu et al.,2023,AgentBench: Evaluating LLMs as Agents,ICLR 2024,conference,arXiv:2308.03688,"https://export.arxiv.org/api/query?id_list=2308.03688 ; https://api.semanticscholar.org/graph/v1/paper/arXiv:2308.03688 (venue='International Conference on Learning Representations', type='conference')",2026-10-30T03:14:02Z,"Multi-environment agent benchmark identifying long-horizon reasoning and instruction following as main failure causes, relevant to protocol adherence measurement.","AgentBench evaluates LLMs as agents across 8 environments; top commercial LLMs act well in complex environments but a significant gap separates them from open-source models up to 70B, with poor long-term reasoning, decision-making and instruction following as main obstacles.",yes46yao2024taubench,5,Yao et al.,2024,$τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains,arXiv preprint,preprint,arXiv:2406.12045,https://export.arxiv.org/api/query?id_list=2406.12045,2026-10-30T03:14:02Z,State-based outcome checking and the pass^k reliability metric are reused in WP29's repeated-run reliability analysis.,"tau-bench evaluates agents in simulated user conversations with API tools and policy guidelines by comparing the final database state to an annotated goal state, and proposes pass^k for reliability over multiple trials; gpt-4o succeeds on fewer than 50% of tasks with pass^8 below 25% in retail.",unknown47kapoor2024agentsthatmatter,5,Kapoor et al.,2024,AI Agents That Matter,Transactions on Machine Learning Research,journal,arXiv:2407.01502,https://export.arxiv.org/api/query?id_list=2407.01502 ; https://api.semanticscholar.org/graph/v1/paper/arXiv:2407.01502 (venue='Trans. Mach. Learn. Res.'),2026-10-30T03:14:03Z,"Core methodological critique (cost reporting, holdouts, conflated benchmarking needs) that WP29's evaluation design follows.","Analysis of agent benchmarks finds a narrow focus on accuracy without cost, leading to needlessly complex and costly agents; proposes jointly optimizing accuracy and cost, and notes that model- and downstream-developer needs are conflated and that many benchmarks have inadequate holdout sets.",yes48sainz2023contamination,5,Sainz et al.,2023,NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark,Findings of the Association for Computational Linguistics: EMNLP 2023,conference,doi:10.18653/v1/2023.findings-emnlp.722,https://api.crossref.org/works/10.18653/v1/2023.findings-emnlp.722 ; doi.org HEAD 302 ; https://export.arxiv.org/api/query?id_list=2310.18018,2026-10-30T03:14:04Z,Defines contamination levels; WP29's task bank must document contamination risk for each task.,"Position paper arguing that training on a benchmark's test split causes overestimation of performance with potentially harmful scientific conclusions, defining levels of data contamination and calling for community measurement for each benchmark.",yes49zheng2023mtbench,5,Zheng et al.,2023,Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,NeurIPS 2023 Datasets and Benchmarks,conference,arXiv:2306.05685,"https://export.arxiv.org/api/query?id_list=2306.05685 ; https://api.semanticscholar.org/graph/v1/paper/arXiv:2306.05685 (venue='Neural Information Processing Systems', type='conference') ; https://api2.openreview.net/notes/search?term=Judging%20LLM-as-a-judge (NeurIPS 2023 Datasets and Benchmarks record)",2026-10-30T03:14:05Z,"Establishes LLM-as-a-judge and its biases (position, verbosity, self-enhancement), which WP29's protected oracles are designed to avoid.","Examines LLM-as-a-judge limitations including position, verbosity and self-enhancement biases, and finds that strong judges such as GPT-4 agree with human preferences over 80% of the time on MT-bench and Chatbot Arena, the same level as human-human agreement.",yes50thakur2025judgingjudges,5,Thakur et al.,2024,Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges,"GEM2 Workshop, ACL 2025",conference,arXiv:2406.12624,"https://export.arxiv.org/api/query?id_list=2406.12624 ; https://api.semanticscholar.org/graph/v1/paper/arXiv:2406.12624 (venue='arXiv.org', type=None) ; arXiv journal_ref: Proceedings of the Fourth Workshop on Generation Evaluation and Metrics GEM2 2025 pages 404 to 430; July 31 August 1 2025; 2025 Association for Computational Linguistics",2026-10-30T03:14:06Z,"Shows even the best judge models lag inter-human agreement, cautioning against unprotected LLM grading of research outputs.","Across thirteen judge models and nine exam-taker models in a high-human-agreement setting, only the best and largest judges reach reasonable alignment with humans, remain well behind inter-human agreement, and may differ from human scores by up to 5 points.",yes51panickssery2024selfpreference,5,Panickssery et al.,2024,LLM Evaluators Recognize and Favor Their Own Generations,NeurIPS 2024,conference,arXiv:2404.13076,"https://export.arxiv.org/api/query?id_list=2404.13076 ; https://api.semanticscholar.org/graph/v1/paper/arXiv:2404.13076 (venue='Neural Information Processing Systems', type='conference')",2026-10-30T03:14:06Z,Self-preference bias is the specific threat WP29 mitigates by separating executor and reviewer and protecting oracles.,"LLM evaluators such as GPT-4 and Llama 2 have non-trivial accuracy at recognizing their own outputs, and fine-tuning reveals a linear correlation between self-recognition capability and the strength of self-preference.",yes52zhu2025abc,5,Zhu et al.,2025,Establishing Best Practices for Building Rigorous Agentic Benchmarks,arXiv preprint,preprint,arXiv:2507.02825,https://export.arxiv.org/api/query?id_list=2507.02825,2026-10-30T03:13:45Z,"Agentic Benchmark Checklist (task validity, outcome validity, reporting) used as a checklist for WP29's task and oracle construction.","Shows that many agentic benchmarks have task-setup or reward-design flaws (e.g., SWE-bench Verified uses insufficient test cases, TAU-bench counts empty responses as successful) that can mis-estimate performance by up to 100% relative, and introduces the Agentic Benchmark Checklist, which reduced overestimation on CVE-Bench by 33%.",no53kapoor2025hal,5,Kapoor et al.,2025,Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation,arXiv preprint,preprint,arXiv:2510.11977,https://export.arxiv.org/api/query?id_list=2510.11977,2026-10-30T03:13:45Z,"Standardized harness with full cost accounting across models, scaffolds and benchmarks, the infrastructure model for WP29's isolated executors and resource logs.","Introduces a standardized evaluation harness and reports 21,730 agent rollouts across 9 models and 9 benchmarks at a total cost of about $40,000, finding among other things that higher reasoning effort can reduce accuracy.",no54li2025llmreval,5,Li et al.,2025,LLM-REVal: Can We Trust LLM Reviewers Yet?,arXiv preprint,preprint,arXiv:2510.12367,https://export.arxiv.org/api/query?id_list=2510.12367,2026-10-30T03:16:08Z,"Simulated research-and-review loop showing LLM reviewers misaligned with humans; motivates WP29's separate, protected review arm rather than in-system AI review.","Using a simulation with a research agent that writes and revises papers and a review agent that assesses them, followed by human annotation, the authors identify pronounced misalignment between LLM reviewers and human judgments.",no55parasuraman1997automation,6,Parasuraman & Riley,1997,"Humans and Automation: Use, Misuse, Disuse, Abuse",Human Factors: The Journal of the Human Factors and Ergonomics Society,journal,doi:10.1518/001872097778543886,https://api.crossref.org/works/10.1518/001872097778543886 ; doi.org HEAD 302,2026-10-30T03:14:09Z,Use/misuse/disuse/abuse taxonomy frames how a researcher delegates to an autonomous agent and over-relies on its outputs.,"Distinguishes use, misuse (over-reliance producing monitoring failures or decision biases), disuse (often caused by false alarms) and abuse of automation, with trust, workload and reliability as key influences.",yes56parasuraman2000levels,6,Parasuraman et al.,2000,A model for types and levels of human interaction with automation,"IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans",journal,doi:10.1109/3468.844354,https://api.crossref.org/works/10.1109/3468.844354 ; doi.org HEAD 302,2026-10-30T03:14:10Z,Levels-of-automation model used to locate WP29's four arms on the acquisition/analysis/decision/implementation dimensions.,"Proposes that automation can be applied to four function classes (information acquisition, information analysis, decision and action selection, action implementation), each across a continuum from fully manual to fully automatic, and that automation changes rather than merely supplants human activity.",yes57goddard2012automationbias,6,Goddard et al.,2012,"Automation bias: a systematic review of frequency, effect mediators, and mitigators",Journal of the American Medical Informatics Association,journal,doi:10.1136/amiajnl-2011-000089,https://api.crossref.org/works/10.1136/amiajnl-2011-000089 ; doi.org HEAD 302,2026-10-30T03:14:10Z,Systematic review of automation bias frequency and mitigators informing how human review of agent outputs is measured in WP29.,"A systematic review (74 of 13,821 retrieved papers) finds automation bias mediated by user factors such as cognitive style, decision-support design and task experience, discussed alongside automation-induced complacency.",yes58lai2023humanai,6,Lai et al.,2023,Towards a Science of Human-AI Decision Making: An Overview of Design Space in Empirical Human-Subject Studies,2023 ACM Conference on Fairness Accountability and Transparency,conference,doi:10.1145/3593013.3594087,https://api.crossref.org/works/10.1145/3593013.3594087 ; doi.org HEAD 302 ; https://export.arxiv.org/api/query?id_list=2112.11471,2026-10-30T03:14:11Z,Design-space survey of empirical human-AI decision-making studies used to position WP29's intervention measurement.,"Surveys empirical human-subject studies of AI-assisted decision making and summarizes the design space of tasks, AI assistance elements and evaluation metrics to structure a science of human-AI decision making.",yes59bansal2021whole,6,Bansal et al.,2021,Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance,Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems,conference,doi:10.1145/3411764.3445717,https://api.crossref.org/works/10.1145/3411764.3445717 ; doi.org HEAD 302,2026-10-30T03:14:11Z,"Shows explanations raise acceptance of AI recommendations regardless of correctness, a risk for human review of agent-generated research.","In studies on three datasets, AI augmentation produced complementary improvements but explanations did not increase them; instead explanations increased the chance that humans accept the AI's recommendation regardless of its correctness.",yes60lubars2019delegability,6,Lubars & Tan,2019,"Ask Not What AI Can Do, But What AI Should Do: Towards a Framework of Task Delegability",NeurIPS 2019,conference,arXiv:1902.03245,"https://export.arxiv.org/api/query?id_list=1902.03245 ; https://api.semanticscholar.org/graph/v1/paper/arXiv:1902.03245 (venue='Neural Information Processing Systems', type='conference')",2026-10-30T03:14:12Z,"Task-delegability framework (motivation, difficulty, risk, trust) used to characterize which research sub-tasks are delegated in each WP29 arm.","Proposes a delegability framework with four factors (motivation, difficulty, risk, trust) and, surveying 100 tasks, finds little preference for full AI control and a strong preference for machine-in-the-loop designs, with trust most correlated with preferences.",yes61vaccaro2024combinations,6,Vaccaro et al.,2024,When combinations of humans and AI are useful: A systematic review and meta-analysis,Nature Human Behaviour,journal,doi:10.1038/s41562-024-02024-1,https://api.crossref.org/works/10.1038/s41562-024-02024-1 ; doi.org HEAD 302,2026-10-30T03:14:12Z,Meta-analytic evidence on when human-AI combinations beat either alone; reference point for interpreting WP29's human-review arm.,"A preregistered meta-analysis of 106 experiments (370 effect sizes, 2020-2023) finds that on average human-AI combinations performed worse than the best of humans or AI alone, with losses in decision tasks and gains in content-creation tasks.",yes62bainbridge1983ironies,6,Bainbridge,1983,Ironies of automation,Automatica,journal,doi:10.1016/0005-1098(83)90046-8,https://api.crossref.org/works/10.1016/0005-1098(83)90046-8 ; doi.org HEAD 302,2026-10-30T03:14:13Z,"Classic argument that automation makes the residual human role harder, relevant to reviewers of autonomous research output.","Argues that the more advanced a control system is, the more crucial the contribution of the human operator becomes, because the operator is left with the tasks the designer could not automate and must monitor a system that rarely needs intervention.",yes63bucinca2021trust,6,Buçinca et al.,2021,To Trust or to Think,Proceedings of the ACM on Human-Computer Interaction,journal,doi:10.1145/3449287,https://api.crossref.org/works/10.1145/3449287 ; doi.org HEAD 302,2026-10-30T03:14:14Z,Cognitive forcing as a mitigation of overreliance; analogous to WP29's requirement that reviewers execute checks rather than read summaries.,"In an experiment with 199 participants, three cognitive forcing interventions reduced overreliance on AI relative to simple explainable-AI approaches.",yes64becker2025metr,6,Becker et al.,2025,Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,arXiv preprint,preprint,arXiv:2507.09089,https://export.arxiv.org/api/query?id_list=2507.09089,2026-10-30T03:14:16Z,Randomized measurement of AI-tool effects on expert developers showing perception-reality gaps; methodological model for WP29's time/cost outcomes.,"In a randomized controlled trial with 16 experienced open-source developers on 246 tasks, allowing early-2025 AI tools increased completion time by 19%, although developers forecast a 24% speedup beforehand and still believed in a 20% speedup afterwards.",no65dean2013tail,7,Dean & Barroso,2013,The tail at scale,Communications of the ACM,journal,doi:10.1145/2408776.2408794,https://api.crossref.org/works/10.1145/2408776.2408794 ; doi.org HEAD 302,2026-10-30T03:14:17Z,"Tail-tolerance techniques (hedging, timeouts) underlie WP29's executor time budgets and retry policy.",Software techniques that tolerate latency variability are vital to building responsive large-scale Web services.,unknown66gray1986whystop,7,Gray,1986,Why Do Computers Stop and What Can Be Done About It?,Symposium on Reliability in Distributed Software and Database Systems,conference,s2:87c9e34c39dd1fbd470010dc1ea98ba3517f5a2a,https://api.semanticscholar.org/graph/v1/paper/87c9e34c39dd1fbd470010dc1ea98ba3517f5a2a (record cached in s2_gray1986_search_observed.json),2026-10-30T03:31:46Z,"Foundational failure analysis (software and operations dominate outages; process pairs, transactions) behind WP29's checkpoint/idempotent step design.","Reports Tandem field data attributing most system outages to software and operations/administration rather than hardware, and advocates process pairs and transactions as the means to tolerate faults.",unknown67basiri2016chaos,7,Basiri et al.,2016,Chaos Engineering,IEEE Software,journal,doi:10.1109/MS.2016.60,https://api.crossref.org/works/10.1109/MS.2016.60 ; doi.org HEAD 302,2026-10-30T03:14:17Z,Chaos engineering as experimentation on failure injection motivates WP29's fault-injection tests of the executors.,"Describes chaos engineering as Netflix's practice of experimenting on distributed systems to verify their reliability, together with the principles underlying it.",yes68helland2012idempotence,7,Helland,2012,Idempotence is not a medical condition,Communications of the ACM,journal,doi:10.1145/2160718.2160734,https://api.crossref.org/works/10.1145/2160718.2160734 ; doi.org HEAD 302,2026-10-30T03:14:18Z,"Idempotent operations make retries safe, a requirement for WP29's resumable protocol steps.",Messages may be retried; idempotence means that is acceptable.,unknown69burckhardt2021durablefunctions,7,Burckhardt et al.,2021,Durable functions: semantics for stateful serverless,Proceedings of the ACM on Programming Languages,journal,doi:10.1145/3485510,https://api.crossref.org/works/10.1145/3485510 ; doi.org HEAD 302,2026-10-30T03:14:20Z,"Formal semantics for durable workflow execution (persistent state, reliable progress) that WP29's run ledger approximates.","Describes Azure Durable Functions, which enhance FaaS with actors, workflows and critical sections to provide persistent state and reliable progress, and gives a formal semantics based on an idealized untyped lambda-calculus model.",yes70erol2025costofpass,7,Erol et al.,2025,Cost-of-Pass: An Economic Framework for Evaluating Language Models,arXiv preprint,preprint,arXiv:2504.13359,https://export.arxiv.org/api/query?id_list=2504.13359,2026-10-30T03:14:21Z,Cost-of-pass (expected cost per correct solution) is the economic outcome metric WP29 reports per arm.,"Defines cost-of-pass as the expected monetary cost of generating a correct solution and the frontier cost-of-pass relative to a human expert, finding lightweight models most cost-effective for basic quantitative tasks and reasoning models for complex quantitative problems.",no71dong2024agentops,7,Dong et al.,2024,AgentOps: Enabling Observability of LLM Agents,arXiv preprint,preprint,arXiv:2411.05285,https://export.arxiv.org/api/query?id_list=2411.05285,2026-10-30T03:14:22Z,Taxonomy of agent-lifecycle artifacts to trace; WP29's evidence records cover these observability artifacts.,"Presents a taxonomy of AgentOps artifacts and data that should be traced throughout the agent lifecycle to achieve observability, derived from a systematic mapping study of existing AgentOps tools.",no72khanal2026reliability,7,Khanal et al.,2026,Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents,arXiv preprint,preprint,arXiv:2603.29231,https://export.arxiv.org/api/query?id_list=2603.29231,2026-10-30T03:15:09Z,"Shows capability (pass@1) and reliability diverge with task duration, supporting WP29's repeated-run reliability outcomes.","Introduces four reliability metrics (Reliability Decay Curve, Variance Amplification Factor, Graceful Degradation Score, Meltdown Onset Point) and, over 23,392 episodes of 10 models on 396 tasks, finds reliability decay is domain-stratified (software-engineering GDS falls from 0.90 to 0.44) while document processing is nearly flat.",no73zhuang2026agentrewind,7,Zhuang et al.,2026,AgentRewind: Recoverable Execution for Long-Horizon LLM Agents,arXiv preprint,preprint,arXiv:2608.14380,https://export.arxiv.org/api/query?id_list=2608.14380,2026-10-30T03:15:09Z,Checkpoint/rewind recovery for long-horizon agents; the failure-recovery mechanism WP29's protocol runs implement at stage boundaries.,"AgentRewind records aligned checkpoints of agent context and a controlled environment so that agents can return to an earlier state and resume with information from previous attempts, evaluated on the MettleBench long-horizon engineering benchmark.",no74sigelman2010dapper,7,Sigelman et al.,2010,"Dapper, a Large-Scale Distributed Systems Tracing Infrastructure",Google Technical Report dapper-2010-1,report,s2:003d5a65de0ac72daaf105ded903cb3eb88585b3,https://api.semanticscholar.org/graph/v1/paper/003d5a65de0ac72daaf105ded903cb3eb88585b3 (record cached in s2_dapper2010.json),2026-10-30T03:38:36Z,"Classic distributed tracing design (sampling, application-level transparency) referenced for WP29's run tracing.","Describes Google's production distributed tracing infrastructure, designed for low overhead through sampling and application-level transparency.",no75wadden2020scifact,8,Wadden et al.,2020,Fact or Fiction: Verifying Scientific Claims,Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),conference,doi:10.18653/v1/2020.emnlp-main.609,https://api.crossref.org/works/10.18653/v1/2020.emnlp-main.609 ; doi.org HEAD 302,2026-10-30T03:14:22Z,Scientific claim verification task and dataset; model for WP29's claim-to-evidence checks in generated reports.,"Introduces scientific claim verification with SciFact, 1.4K expert-written claims paired with evidence abstracts annotated with labels and rationales; the system identified plausible evidence for 23 of 36 COVID-19 claims on CORD-19.",yes76thorne2018fever,8,Thorne et al.,2018,FEVER: a Large-scale Dataset for Fact Extraction and VERification,"Proceedings of the 2018 Conference of the North American Chapter of77 the Association for Computational Linguistics: Human Language78 Technologies, Volume 1 (Long Papers)",conference,doi:10.18653/v1/N18-1074,https://api.crossref.org/works/10.18653/v1/N18-1074 ; doi.org HEAD 302,2026-10-30T03:14:23Z,Canonical fact extraction and verification benchmark defining Supported/Refuted/NotEnoughInfo labels adopted in WP29's citation audit.,"FEVER contains 185,445 claims derived from Wikipedia and labeled Supported, Refuted or NotEnoughInfo (Fleiss kappa 0.6841), with a best pipeline accuracy of 31.87% when the correct evidence is required.",yes79bohnet2022attributedqa,8,Bohnet et al.,2022,Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models,arXiv preprint,preprint,arXiv:2212.08037,https://export.arxiv.org/api/query?id_list=2212.08037,2026-10-30T03:14:23Z,Attributed QA frames generation that must cite supporting sources; WP29 requires attributed claims in agent reports.,"Formulates Attributed QA with a reproducible evaluation framework that uses human annotations as gold standard and shows that a correlated automatic metric is suitable for development, benchmarking a broad set of architectures.",no80walters2023fabrication,8,Walters & Wilder,2023,Fabrication and errors in the bibliographic citations generated by ChatGPT,Scientific Reports,journal,doi:10.1038/s41598-023-41032-5,https://api.crossref.org/works/10.1038/s41598-023-41032-5 ; doi.org HEAD 302,2026-10-30T03:14:24Z,Quantifies fabricated references in LLM literature reviews; justifies WP29's mandatory citation verification gate.,"In 84 short literature reviews on 42 topics containing 636 citations, 55% of GPT-3.5 citations and 18% of GPT-4 citations were fabricated, and 43% versus 24% of the real citations contained substantive errors.",yes81alkaissi2023hallucinations,8,Alkaissi & McFarlane,2023,Artificial Hallucinations in ChatGPT: Implications in Scientific Writing,Cureus,journal,doi:10.7759/cureus.35179,https://api.crossref.org/works/10.7759/cureus.35179 ; doi.org HEAD 302,2026-10-30T03:14:24Z,Early clinical report of fabricated references ('artificial hallucinations') in scientific writing.,"Reports two case studies in which ChatGPT produced plausible but fabricated content including references when asked to write on rare conditions, documenting positive, negative and troubling aspects of its performance.",unknown82gao2023rarr,8,Gao et al.,2023,"RARR: Researching and Revising What Language Models Say, Using Language Models",Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),conference,doi:10.18653/v1/2023.acl-long.910,https://api.crossref.org/works/10.18653/v1/2023.acl-long.910 ; doi.org HEAD 302,2026-10-30T03:14:25Z,Post-hoc attribution and revision pipeline; a model for WP29's independent evidence-grounding pass over agent output.,"RARR automatically finds attribution for the output of any text generator and post-edits unsupported content, significantly improving attribution while otherwise preserving the original output.",yes83wang2023selfconsistency,8,Wang et al.,2022,Self-Consistency Improves Chain of Thought Reasoning in Language Models,ICLR 2023,conference,arXiv:2203.11171,"https://export.arxiv.org/api/query?id_list=2203.11171 ; https://api.semanticscholar.org/graph/v1/paper/arXiv:2203.11171 (venue='International Conference on Learning Representations', type='conference')",2026-10-30T03:14:25Z,Self-consistency is an in-model verification heuristic; WP29 contrasts it with external executable verification.,"Self-consistency samples diverse reasoning paths and selects the most consistent answer, improving chain-of-thought accuracy on GSM8K (+17.9%), SVAMP (+11.0%) and AQuA (+12.2%).",yes84min2023factscore,8,Min et al.,2023,FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,conference,doi:10.18653/v1/2023.emnlp-main.741,https://api.crossref.org/works/10.18653/v1/2023.emnlp-main.741 ; doi.org HEAD 302 ; https://export.arxiv.org/api/query?id_list=2305.14251,2026-10-30T03:14:25Z,Atomic-fact precision scoring is the template for WP29's claim-level verification of generated reports.,"FActScore decomposes a generation into atomic facts and computes the percentage supported by a reliable knowledge source; ChatGPT biographies reach only 58%, and an automated estimator is introduced.",yes85huang2024selfcorrect,8,Huang et al.,2023,Large Language Models Cannot Self-Correct Reasoning Yet,ICLR 2024,conference,arXiv:2310.01798,"https://export.arxiv.org/api/query?id_list=2310.01798 ; https://api.semanticscholar.org/graph/v1/paper/arXiv:2310.01798 (venue='International Conference on Learning Representations', type='conference')",2026-10-30T03:14:26Z,"Evidence that intrinsic self-correction fails without external feedback, supporting WP29's external gates over agent self-review.","Finds that LLMs struggle to self-correct their reasoning without external feedback, and that performance can even degrade after intrinsic self-correction.",yes86sakai2026hallucitation,8,Sakai et al.,2026,HalluCitation Matters: Revealing the Impact of Hallucinated References with 300 Hallucinated Papers in ACL Conferences,ACL 2026 (per Semantic Scholar venue record),conference,arXiv:2601.18724,"https://export.arxiv.org/api/query?id_list=2601.18724 ; https://api.semanticscholar.org/graph/v1/paper/arXiv:2601.18724 (venue='Annual Meeting of the Association for Computational Linguistics', type='conference')",2026-10-30T03:15:08Z,"Documents hallucinated citations reaching accepted NLP papers, showing the scale of the problem WP29's citation gate addresses.","Analyzing all ACL, NAACL and EMNLP papers of 2024-2025, nearly 300 papers contain at least one hallucinated citation, half of them at EMNLP 2025, and more than 100 such papers were accepted as main-conference or Findings papers.",yes87zhao2026hallucinationswild,8,Zhao et al.,2026,LLM hallucinations in the wild: Large-scale evidence from non-existent citations,arXiv preprint,preprint,arXiv:2605.07723,https://export.arxiv.org/api/query?id_list=2605.07723,2026-10-30T03:15:08Z,Large-scale audit of non-existent citations in the literature after LLM adoption; external evidence of the citation-fabrication risk.,"Auditing 111 million references across 2.5 million papers in arXiv, bioRxiv, SSRN and PubMed Central shows a sharp rise in non-existent references after widespread LLM adoption, with a conservative estimate of 146,932 hallucinated citations in 2025 alone.",no88