system,ref_key,inputs,workflow_structure,verification_mechanism,human_involvement,task_scope,resources_reported,evaluated_outcomes,comparison_to_pgar_design
The AI Scientist (v1; Nature 2026 version),lu2024aiscientist; lu2026aiscientist,"Human-written code template plus seed task in an ML subfield (diffusion, language modeling, learning dynamics).",Fixed pipeline: idea generation -> novelty check (Semantic Scholar) -> code editing/experiments (Aider) -> plots -> paper writing -> simulated review; open-ended iteration in principle.,Automated LLM reviewer trained/validated against ICLR review data; experiment success is whether code runs; no executable scientific-validity gates reported.,None during the run (fully automated); humans provide templates and read outputs.,Machine-learning research in three subfields with provided templates.,Less than $15 per paper (arXiv abstract).,Paper quality as scored by the automated reviewer (claimed to exceed a top-conference acceptance threshold); near-human agreement of the reviewer with human scores.,"Established: an end-to-end pipeline can produce reviewable ML papers cheaply. Not established: whether structure (protocol vs open instruction) changes validity, because there is a single pipeline, the reviewer is in-system (self-preference risk, Panickssery et al. 2024) and there is no independent oracle. PGAR differs by holding the executor fixed and randomizing the instruction structure across four arms, with oracles held outside the executor's reach."
The AI Scientist-v2,yamada2025aiscientistv2,Research topic prompt; no human code template.,"Progressive agentic tree search over experiment nodes managed by an experiment-manager agent; stages: preliminary investigation, hyperparameter tuning, research agenda execution, ablations; then paper writing.",In-system AI reviewer with VLM feedback on figures; node success judged by execution and metrics; external check was one ICLR workshop submission that passed review (score above average acceptance threshold).,"Minimal (topic selection, submission); the workshop reviewers were human but unaware of which papers were AI-generated for most of the process (per the preprint).",ML workshop-level research across diverse ML domains.,Not reported in the abstract (compute/cost per manuscript not given there).,Workshop acceptance of 1 of 3 manuscripts; qualitative review scores.,"Established: tree search plus self-review can reach workshop-acceptance scores. Not established: how much of that depends on verification structure; Luo et al. (2025) found leakage/metric-misuse/post-hoc selection in such systems and Beel et al. (2025) found novelty errors in v1. PGAR's gates are executable, external and identical across executors, and its outcome is scientific validity graded by protected oracles rather than reviewer score."
Agent Laboratory,schmidgall2025agentlab,Human-provided research idea; optional feedback at each stage.,"Three stages: literature review, experimentation (mle-solver), report writing (paper-solver); copilot vs autonomous modes.",Human evaluation survey of outputs; automated reviewer scores; code quality judged by task performance; no executable scientific-validity gates.,Explicit co-pilot mode: humans give feedback at each stage; paper reports that this significantly improves quality.,ML research tasks given an idea.,84% lower cost than previous autonomous methods (abstract); absolute dollar figures in the paper not re-read here.,"Human ratings of quality/usefulness, automated review scores, cost.","Already established: human feedback at stage boundaries improves outcomes and staging lowers cost. PGAR keeps the human out of the loop during execution and instead varies whether the agent receives a protocol, gates and evidence requirements; separate review is by a protected evaluator, not by the same system. The comparison is therefore about instruction structure, not about human co-piloting."
Coscientist,boiko2023coscientist,"Natural-language experimental goal; access to web search, documentation, code execution, cloud lab API.","LLM planner with tool modules (web, docs, code, automation) iterating plan-act-observe.","Physical/experimental outcome (e.g., reaction yields) and author inspection; no automated scientific-validity gate.",Humans design tasks and supervise/approve physical execution.,Six chemistry tasks including reaction optimization.,Not reported (in abstract).,Success of planned experiments and optimization performance.,"Established: an LLM agent can plan and run real experiments. Not comparable on the structural question: single system, single arm, physical oracle. PGAR's oracles are computational and hidden, and the manipulated variable is the protocol itself."
"AI co-scientist (Co-Scientist, Nature 2026)",gottweis2026coscientist,Research goal in natural language plus prior evidence.,Multi-agent generate / reflect / rank (tournament) / evolve loop with asynchronous task execution and test-time compute scaling; outputs hypotheses and proposals.,In-system critique and tournament ranking; downstream wet-lab validation of selected hypotheses by collaborators.,"Scientists specify goals, interact, and run experimental validation.",Hypothesis generation in biomedicine.,Compute scaling described qualitatively; cost not reported in abstract.,"Novelty/quality of hypotheses, expert ratings, experimental confirmation of selected hypotheses.","Established: self-critique loops improve hypothesis quality under expert evaluation. Not addressed: executing analyses, producing verifiable artifacts, or contrasting protocol structures. PGAR is about executing a complete computational research task under alternative instruction regimes with external verification."
data-to-paper,ifargan2025datatopaper,Annotated dataset plus research goal (or open goal).,"Stepwise multi-agent pipeline (data exploration, hypothesis, analysis code, results, manuscript) with programmatic back-tracing of every number to code/data.",Traceability chain ('chain of information') enabling human verification; autopilot vs copilot modes; authors re-examined outputs for errors.,Optional human intervention at steps (copilot); autopilot is fully autonomous.,Data-driven papers from tabular datasets (public health / biomedical examples).,Not reported in abstract.,Correctness/verifiability of produced papers as assessed by the authors; number of papers generated and error analysis.,"Closest prior on persistent evidence records: back-tracing information flow is a form of evidence ledger. Differences: a single pipeline (no control arm without traceability), human-verifiable rather than machine-executed gates, and no protected oracle. PGAR tests whether such records plus gates change validity relative to an open instruction with the same executor."
Virtual Lab,swanson2025virtuallab,Research goal; human PI oversight.,"LLM principal investigator agent runs team meetings with specialist LLM agents and a critic agent, then individual meetings produce code/designs.",Scientific-critic agent in-loop; final experimental validation of designed nanobodies in the wet lab.,Human researcher guides agendas and performs experiments.,Nanobody design (computational biology).,Not reported in abstract.,Experimental binding results for designed candidates.,"Established: structured agent meetings with a critic plus human oversight can yield validated designs. Not a controlled comparison of structures; the critic is in-system. PGAR isolates the executor and uses external gates/oracles, and compares structures."
Kosmos,mitchener2025kosmos,Open-ended objective plus dataset.,"Up to 12-hour runs: parallel data-analysis and literature-search agents share a structured world model; cycles of analysis, search, hypothesis; report synthesis.",Citations to code/literature in reports; accuracy of statements assessed by expert review (per preprint).,None during run; humans read reports.,Data-driven discovery across scientific domains.,Up to 12 h and ~200 agent rollouts per run (abstract); cost not in abstract.,Expert-judged accuracy of report statements; case-study discoveries.,Established: long-horizon coherence via a world model. Not established: whether a fixed protocol with executable gates changes validity. PGAR's design is shorter-horizon but controlled and replicated across arms.
XScientist (git-like research protocol),luo2026xscientist,Research question; agent-native research artifact (ARA) state.,"Typed content-addressed objects (hypotheses, attempts, observations, claims, reviews, handoffs) in an exploration graph; immutable checkpoints; replay boundaries; authority-aware gates.",Claim-evidence closure checks and integrity gates; the authors state a passing integrity check is not scientific truth.,Supervised long-run operation; human or agent can fork/inspect/extend.,Long-running autonomous discovery (general).,Not reported in abstract.,Protocol/artifact properties; reference implementation; no controlled validity experiment reported in abstract.,"Closest conceptual neighbor to PGAR's protocol + evidence records (July 2026). Difference: XScientist is a protocol artifact and implementation, not an experiment; PGAR contributes the controlled four-arm comparison measuring what protocol structure buys, with isolated executors and protected oracles. PGAR should cite it as concurrent, independent work."
ScienceAgentBench,chen2025scienceagentbench,"Task instruction, dataset, optional expert knowledge; output is a self-contained Python program.","Benchmark (102 tasks from 44 papers); agents run with direct prompting, OpenHands CodeAct or self-debug.","Program execution, result comparison metrics, figure judging; cost tracked; contamination mitigations.",Experts validate tasks; no human in agent loop.,Data-driven discovery tasks in four disciplines.,API cost per task reported; o1-preview more than 10x cost.,"Success rate (32.4% best; 42.2% o1-preview), cost.","Established: per-task evaluation with execution-based oracles and cost. PGAR reuses the idea of executable oracles but manipulates the instruction regime rather than the model/framework, and adds persistent evidence records and separate review as treatments."
DiscoveryBench,majumder2025discoverybench,"Dataset, metadata, natural-language discovery goal.",Benchmark of 264 real + 903 synthetic multi-step discovery tasks; agents produce hypotheses.,Hypothesis matching against gold (structured context/variables/relationships).,None in loop; tasks derived from papers by annotators.,Data-driven discovery across six domains.,Not reported in abstract.,Hypothesis-level correctness.,Established: formalization of discovery as a task unit. PGAR's unit is similar but the outcome is full-report scientific validity and the treatment is protocol structure.
MLAgentBench,huang2024mlagentbench,Task description plus starter code/data.,13 ML experimentation tasks; ReAct agent with file and code actions.,Execution and performance metric improvement.,None.,ML experimentation.,Not reported in abstract.,Success rate by model.,Established: executors can be benchmarked on experimentation loops. No protocol manipulation.
BLADE,gu2024blade,Dataset + research question.,"Benchmark; agents choose variables, transformations, models.",Automatic matching of analysis decisions against independent expert analyses (multiple valid paths).,Experts provide ground truth; none in loop.,Open-ended data-driven science questions (12).,Not reported in abstract.,Decision-level agreement with experts.,Established: multi-path ground truth for analysis decisions. PGAR's oracles must similarly accept multiple valid analyses; PGAR adds the structural manipulation.
CORE-Bench,siegel2024corebench,Paper's code and data.,Benchmark of 270 reproduction tasks (three difficulty levels).,Question answering about reproduced results checked against ground truth.,None.,Computational reproducibility of 90 papers.,Reported in paper (cost tracked per agent run; figures not re-read here).,Accuracy by difficulty.,Established: reproduction as an executable oracle. PGAR's release/evidence gate is informed by this but PGAR evaluates de novo research under alternative protocols.
PaperBench,starace2025paperbench,Paper text; agent must replicate from scratch.,"Benchmark of 20 ICML 2024 papers decomposed into 8,316 rubric items.",Rubric grading by an LLM judge validated on a separate judge benchmark (JudgeEval).,Rubrics co-developed with authors; none in loop.,AI research replication.,Reported in paper (not re-read here).,Rubric-weighted replication score.,Established: hierarchical rubrics plus validated LLM judging. PGAR adopts validated rubrics/oracles but keeps them protected from the executor and compares protocol arms.
AI Agents That Matter / HAL,kapoor2024agentsthatmatter; kapoor2025hal,n/a (methodology and harness).,"Argues for cost-controlled, held-out, reproducible agent evaluation; HAL harness runs 21,730 rollouts.",Standardized harness; cost accounting; held-out sets.,n/a,Agent benchmarking in general.,"HAL: ~$40,000 for 21,730 rollouts (abstract).",Accuracy-cost Pareto; reproducibility of claims.,"Established: evaluation methodology PGAR follows (cost, holdouts, standardized harness). Not a study of research-protocol structure."
Agentic Benchmark Checklist (ABC),zhu2025abc,n/a (checklist).,"Task validity, outcome validity, reporting checklist.","Audit of reward designs (e.g., empty-response success).",n/a,Agentic benchmarks.,n/a,Overestimation reduced by 33% on CVE-Bench.,PGAR applies ABC to its own tasks/oracles; ABC does not study protocol structure.
"Hidden pitfalls audit (Luo, Kasirzadeh, Shah 2025) and Beel et al. 2025 evaluation",luo2025pitfalls; beel2025sakana,Existing open-source AI scientist systems.,"Controlled experiments isolating four failure modes (benchmark selection, leakage, metric misuse, post-hoc selection); independent re-run of the AI Scientist.","Inspection of traces, code and papers by the auditors.",Auditors only.,AI scientist systems.,Not reported (abstracts).,Presence of failure modes; novelty errors; execution failures.,"These are audits of existing systems, establishing that failures exist and are visible in traces rather than papers. PGAR is the next step: a controlled test of whether protocol structure, gates and evidence records reduce those failures, measured by protected oracles rather than post-hoc audit."