1{"ts": "2026-11-03T23:51:12+00:00", "task": "draft_introduction", "doc": "wp29", "domain": "out_of_domain", "condition": "C_conventional_rag", "rep": 1, "paper": "/Users/simon-pierreboucher/Desktop/UQO/Recherche/Recherche-UQO/methodes/autonomous-research-claude-code", "degraded_literature": "", "ok": true, "text": "\\label{sec:intro}\n\nResearchers who hand a scientific task to a coding agent must decide how to write the hand-off. In practice, the instruction files that steer such agents range from a single sentence to several thousand words of phases, gates, ledgers and integrity rules, and the longer ones resemble preregistrations. This paper asks whether that structure changes what the agent delivers, and at what cost, when the agent itself is held fixed.\n\nThe question matters because the evidence on autonomous research points in two directions. End-to-end systems built on large language models now produce manuscripts at low cost and with reviewer scores near acceptance thresholds \\citep{lu2024aiscientist,yamada2025aiscientistv2,ifargan2025datatopaper}. Independent audits of the same systems find data leakage, metric misuse, post-hoc selection and inflated novelty claims. These problems are visible in execution traces but not in the papers \\citep{luo2025pitfalls,beel2025sakana}. A natural response is to constrain the agent with an explicit protocol. Whether such constraints improve validity, however, cannot be read from evaluations of single pipelines without a control arm, in which the instruction, the scaffold and the degree of human co-piloting move together.\n\nWe isolate the instruction regime. We ran \\nMainRuns{} live Claude Code runs (\\nMainSessions{} sessions; client 2.1.314, model \\texttt{claude-opus-5-5}) on \\nConfTasks{} bounded scientific tasks in six families: data audit, estimation and inference, forecasting under information constraints, source-grounded synthesis, theory and numerical validation, and integrated mini-projects. Each task received \\nRepeats{} fresh sessions under each of four regimes:\n\\begin{itemize}\n\\item[(A)] an open instruction;\n\\item[(B)] a structured protocol with phases, ledgers, stopping and recovery rules;\n\\item[(C)] the protocol plus a controller-enforced executable gate that checks the deliverable contract and reproducibility and returns failures to the session;\n\\item[(D)] the same plus a separate review-and-repair session.\n\\end{itemize}\nExecutors ran in bare mode on nodes that held no copy of the reference answers. A deterministic, task-specific evaluator scored every run on a separate host; each evaluator had been validated beforehand on seeded correct and incorrect artefacts. The design, tasks and analysis were frozen with hashes before the confirmatory runs. The primary endpoint is intervention-free verified completion within a shared resource envelope. The primary contrast is D$-$A, with inference at the level of the \\nConfTasks{} task clusters.\n\nThe answer is a precise null. Completion was $\\pA$ under the open instruction and $\\pD$ under the full method. The paired difference is $\\dDA$, with a task-cluster bootstrap 95\\% interval of $[\\dDAlo, \\dDAhi]$. The mechanism contrasts B$-$A, C$-$B and D$-$C are equally flat, and critical integrity violations were rare in every arm.\n\nThe added structure changed two things. First, contract compliance: no arm-C or arm-D deliverable failed the post-hoc contract check, whereas \\nContractFailA{} arm-A and \\nContractFailB{} arm-B deliverables did. Second, cost: inference cost per run rose by factors of \\costRatioB{}, \\costRatioC{} and \\costRatioD{} for arms B, C and D relative to arm A. Two qualifications bound the comparison. The review-and-repair branch of arm D was implemented but never triggered. A launcher defect also truncated every continuation prompt (amendment A-006), so the frozen comparison is in effect one between protocol texts delivered through the system prompt.\n\nThe failures are more informative than the contrasts. On inspection, every non-success under the frozen scoring traces to a defect of our oracles rather than of the executors. In one task, the source publishes no price index for one window month. All twelve sessions correctly reported the growth rate as undefined, but the oracle required a feasibility label that the task never defined. In another, the data generator had produced impossible negative training hours. Eleven of twelve sessions excluded these values with a documented ledger, while the oracle's reference used all 1,200 rows. The remaining two failures were an unjustified rounding tolerance and a regular-expression false positive. Under a corrected rubric, every arm completes every task, and the null is unchanged.\n\nWe do not read these results as evidence that research protocols are useless. We read them as evidence about where the difficulty of delegated research lies for a current frontier model on bounded tasks. The difficulty is not in organising the work, which the model does competently without being told. It is in knowing what the right answer is. In this study, the binding cost of verification was not the agent's inference budget. It was the construction of tasks with known answers, oracles the executor cannot read, and checks that do not trust an exit code. The same checks caught harness defects that would otherwise have produced confident conclusions in either direction.\n\nThe paper makes four contributions. First, we formalise delegation as a contract and provide a small reference implementation in which every claimed invariant is enforced by code. The implementation comprises a controller, durable state, isolated executors, an oracle-free runtime gate, separate review and hidden evaluators. Second, we construct a benchmark of \\nConfTasks{} confirmatory and \\nDevTasks{} development tasks with hidden oracles and seeded validation sets, including \\nConfRealData{} real-data and \\nConfSourceDocs{} source-document tasks, plus three held-out transfer mini-projects. Third, we report what is, to our knowledge, the first preregistered comparison of instruction regimes in which the agent, tools, inputs and budget are held fixed and the oracles are protected from the executor. It complements benchmarks that score agents on individual tasks \\citep{chen2025scienceagentbench} and protocol designs proposed without an experiment \\citep{luo2026xscientist}. Fourth, we provide an instrumented account of how this paper was itself produced under a mandate of the same kind, kept separate from the confirmatory evidence.\n\nSection~\\ref{sec:related} positions the work. Sections~\\ref{sec:framework}--\\ref{sec:evaluation} describe the framework, the corpus of researcher-written mandates that motivated it, the implementation and the design. Sections~\\ref{sec:results}--\\ref{sec:transfer} report the main results, the recovery, fault and ablation experiments, and the transfer projects. Section~\\ref{sec:reflexive} gives the reflexive account, and Section~\\ref{sec:limitations} states what the study cannot establish.", "change_log": "- Reordered the section to follow a standard economics introduction: research question first, then motivation from the two opposing literatures, design, magnitudes, interpretation, contributions and roadmap.\n- Split the dense design and results paragraphs. Each paragraph now carries one idea, and the four arms are set out as a short list for readability.\n- Shortened the failure-analysis paragraph to the four oracle defects. Harness-defect detail (wrong-workspace evaluator, A-004 ledger collision) is left to Section~\\ref{sec:results}.\n- Moved the closest-prior-work positioning (ScienceAgentBench, XScientist) into contribution (iii) rather than writing a separate literature paragraph.\n- Used the manuscript's macros for all magnitudes. Literal numbers (twelve, eleven of twelve, 1,200, client 2.1.314) appear verbatim in the material.\n- Length is roughly 950 words, close to the 848-word target.", "flags": "- \"Price index\" is a slight generalisation of \"consumer-price index for October 2025\" (Main results). Restore the specific wording if preferred.\n- \"To our knowledge, the first\" preregistered comparison is retained from the author's draft and should be confirmed against the full related-work section.\n- The roadmap assumes labels sec:evaluation, sec:recovery, sec:transfer, sec:reflexive and sec:limitations exist. Only some of them appear in the supplied text.\n- The contract-failure counts (\\nContractFailA{}, \\nContractFailB{}) and cost ratios (\\costRatioB{}, \\costRatioC{}, \\costRatioD{}) are macros whose values were not visible. Check that the cost ratios are consistent with the \"two- to three-fold\" wording in the abstract.", "cost_usd": 0.12755, "latency_s": 29.6, "usage": {"input": 14753, "output": 3427}, "n_calls": 1, "model": "claude-opus-5-5", "effort": "medium", "metrics": {"n_words": 939, "preserved_rate": 0.068, "new_numbers": 0, "new_rate": 0.0, "n_author_year": 0, "n_unverifiable_citations": 0, "unverifiable_examples": [], "unknown_cite_keys": 0, "n_unsupported_claims": 1, "max_overlap_share": 0.0, "todo_placeholders": 0}, "retrieved": [{"unit": "section_opening", "score": 0.9878, "doc_id": "jof_12766", "doi": "10.1111/jofi.12766", "title": "Labor-Technology Substitution: Implications for Asset Pricing", "year": 2019, "page": 1, "section": "Introduction (unnumbered)", "sha256": "2f9e655c4aaab4f1", "role": "introduction", "genre": "mixed_theory_empirical", "subfield": "asset_pricing", "confidence": 0.963, "split": "train"}, {"unit": "section_opening", "score": 0.7346, "doc_id": "jof_12779", "doi": "10.1111/jofi.12779", "title": "Proxy Advisory Firms: The Economics of Selling Information to Voters", "year": 2019, "page": 1, "section": "Introduction (unnumbered)", "sha256": "52ff2e9b20d30f3d", "role": "introduction", "genre": "theoretical", "subfield": "corporate_finance", "confidence": 0.938, "split": "train"}, {"unit": "section_opening", "score": 0.7236, "doc_id": "jof_12619", "doi": "10.1111/jofi.12619", "title": "What Makes a Good Trader? On the Role of Intuition and Reflection on Trader Performance", "year": 2018, "page": 1, "section": "Introduction (unnumbered)", "sha256": "034e718ad966ca13", "role": "introduction", "genre": "empirical", "subfield": "behavioral_finance", "confidence": 0.938, "split": "train"}, {"unit": "section_opening", "score": 0.6703, "doc_id": "jof_12509", "doi": "10.1111/jofi.12509", "title": "Do Funds Make More When They Trade More?", "year": 2017, "page": 1, "section": "Introduction (unnumbered)", "sha256": "d97f9e3543f798e3", "role": "introduction", "genre": "mixed_theory_empirical", "subfield": "asset_pricing", "confidence": 0.963, "split": "train"}, {"unit": "section_opening", "score": 0.6689, "doc_id": "jof_12856", "doi": "10.1111/jofi.12856", "title": "The Insurance Is the Lemon: Failing to Index Contracts", "year": 2020, "page": 1, "section": "Introduction (unnumbered)", "sha256": "9c4ffadde6f75d24", "role": "introduction", "genre": "theoretical", "subfield": "corporate_finance", "confidence": 0.963, "split": "train"}, {"unit": "section_opening", "score": 0.6667, "doc_id": "jof_12591", "doi": "10.1111/jofi.12591", "title": "Measuring Liquidity Mismatch in the Banking Sector", "year": 2018, "page": 1, "section": "Introduction (unnumbered)", "sha256": "2743beca9bf99d3d", "role": "introduction", "genre": "mixed_theory_empirical", "subfield": "financial_intermediation_banking", "confidence": 0.963, "split": "train"}], "profile": null}2{"ts": "2026-11-03T23:51:13+00:00", "task": "draft_introduction", "doc": "wp29", "domain": "out_of_domain", "condition": "B_strong_prompt", "rep": 1, "paper": "/Users/simon-pierreboucher/Desktop/UQO/Recherche/Recherche-UQO/methodes/autonomous-research-claude-code", "degraded_literature": "", "ok": true, "text": "\\label{sec:intro}\n\nResearchers who delegate work to coding agents write instructions for them, and those instructions vary widely. Some are a single sentence. Others run to several thousand words of phases, gates, ledgers and integrity rules that read like preregistrations. This paper asks whether that structure changes what the agent delivers, and at what cost, when the agent itself is held fixed.\n\nThe question matters because the evidence on autonomous research points in two directions. End-to-end systems now produce manuscripts at low cost, with reviewer scores near acceptance thresholds \\citep{lu2024aiscientist,yamada2025aiscientistv2,ifargan2025datatopaper}. Independent audits of such systems find data leakage, metric misuse, post-hoc selection and unfounded novelty claims, all visible in the execution traces but not in the papers \\citep{luo2025pitfalls,beel2025sakana}. Recent proposals respond with more structure: typed research objects, claim--evidence closure and integrity gates \\citep{luo2026xscientist}. Yet each existing system is a single pipeline without a control arm, so none can separate the effect of the instruction regime from the effects of the model, the scaffold or human co-piloting. Task-level benchmarks supply executable oracles \\citep{chen2025scienceagentbench}, but they hold the instruction fixed and vary the agent, which is the opposite of what the question requires.\n\nWe vary the instruction and hold everything else fixed. We ran \\nMainRuns{} live Claude Code runs (\\nMainSessions{} sessions; client 2.1.314, model \\texttt{claude-opus-5-5}) on \\nConfTasks{} bounded scientific tasks in six families: data audit, estimation and inference, forecasting under information constraints, source-grounded synthesis, theory and numerical validation, and integrated mini-projects. Each task received \\nRepeats{} fresh sessions under each of four regimes:\n\\begin{itemize}\n\\item an open instruction (A);\n\\item a structured protocol with phases, ledgers, stopping and recovery rules (B);\n\\item the protocol plus a controller-enforced executable gate that checks the deliverable contract and reproducibility and returns failures to the session (C);\n\\item the same plus a separate review-and-repair session (D).\n\\end{itemize}\nExecutors ran in bare mode on hosts that held no copy of the reference answers. A deterministic, task-specific evaluator scored every run on a separate host; each evaluator was first validated on seeded correct and incorrect artefacts. The design, tasks and analysis were frozen with hashes before the confirmatory runs. The primary endpoint is intervention-free verified completion within a shared resource envelope. The primary contrast is D$-$A, with bootstrap intervals and permutation tests clustered at the task level.\n\nStructure did not change completion. Intervention-free verified completion was $\\pA$ under the open instruction and $\\pD$ under the full method. The paired task-level difference is $\\dDA$, with a 95\\% interval of $[\\dDAlo, \\dDAhi]$. The mechanism contrasts B$-$A, C$-$B and D$-$C are equally flat, and critical integrity violations were rare in every arm. Structure did change two other things. First, no arm C or D deliverable failed the post-hoc contract check, whereas \\nContractFailA{} arm-A and \\nContractFailB{} arm-B deliverables did. Second, inference cost per run rose two- to three-fold. A length-matched unstructured control, interruption-and-resumption experiments, stale-input, duplicate-job and missing-fixture faults, and three held-out end-to-end mini-projects complete the evidence (Sections~\\ref{sec:recovery} and~\\ref{sec:transfer}).\n\nTwo features of the implementation bound what the comparison tests. The review-and-repair branch of arm D was available but never triggered. A launcher defect truncated every continuation prompt (amendment A-006), so the frozen comparison is, in effect, one of protocol texts delivered through the system prompt.\n\nInspection of the failures shows where the difficulty lies. Every non-success under the frozen scoring traces to a defect in our oracles rather than in the executors. In one task the source publishes no price index for one month, and the sessions reported the affected growth rate as undefined, which the oracle did not allow. In another, the data generator produced impossible negative training hours, and sessions excluded them with a documented ledger while the oracle's reference kept them. The two isolated failures were a rounding tolerance and a regular-expression false positive. Under a corrected rubric every arm completes every task, and the null is unchanged.\n\nFor a current frontier model on bounded tasks, the hard part of delegated research was therefore not organising the work, which the model did competently without being told how. The hard part was knowing what the right answer is. In this study the binding cost of verification was not the agents' inference budget. It was the construction and validation of tasks with known answers, oracles the executor cannot read, and checks that do not trust an exit code. Harness defects caught along the way, such as an evaluator pointed at the wrong workspace and a ledger key collision (amendment A-004), would otherwise have produced confident conclusions in either direction.\n\nThe paper makes four contributions:\n\\begin{enumerate}\n\\item A formalisation of delegation as a contract, and a small reference implementation in which every claimed invariant is enforced by code and every limit is stated.\n\\item A benchmark of \\nConfTasks{} confirmatory and \\nDevTasks{} development tasks with hidden oracles and seeded validation sets, including \\nConfRealData{} real-data and \\nConfSourceDocs{} source-document tasks with provenance, plus three transfer mini-projects.\n\\item To our knowledge, the first preregistered comparison of instruction regimes that holds the executing agent, tools, inputs and budget fixed and protects the oracles from the agent.\n\\item An instrumented account of how this paper was itself produced under a mandate of the same kind, kept separate from the confirmatory evidence.\n\\end{enumerate}\n\nSection~\\ref{sec:related} positions the work. Sections~\\ref{sec:framework}--\\ref{sec:evaluation} present the framework, the corpus of researcher-written mandates that motivated it, the implementation and the design. Sections~\\ref{sec:results}--\\ref{sec:transfer} report results. Section~\\ref{sec:reflexive} gives the reflexive account, and Section~\\ref{sec:limitations} states what the study cannot establish.", "change_log": "- Reordered so the research question comes first and the literature follows as motivation. The prior work is reduced to one argument: single pipelines without control arms, and benchmarks that vary the agent rather than the instruction.\n- Replaced \"precise null\" with plain statements of what structure did and did not change, which keeps the claim proportionate.\n- Moved the arm D and A-006 caveats into their own short paragraph so they qualify the result without burying it.\n- Recast the arms and the contributions as lists for readability.\n- Used only manuscript macros and facts (\\pA, \\dDA, \\nContractFailA, A-004/A-006, client and model versions, the six families). No new numbers or citations were added.\n- Dropped the \"about three/six percentage points\" phrasing so the text relies on the interval macros.", "flags": "- Possible inconsistency. The abstract says \"twelve sessions correctly excluded\" the negative training hours, but Main results says eleven of twelve did so (the twelfth kept all rows and passed). I avoided giving a count; please reconcile the abstract.\n- The \"first ... to our knowledge\" claim in contribution 3 needs confirmation against Related work, especially \\citet{luo2026xscientist} and \\citet{ding2026verificationgap}.\n- The claim that the evaluator was \"pointed at the wrong workspace\" is carried over from the existing introduction. I found no amendment number for it in the supplied material, so please confirm and cite the relevant amendment.\n- Verify that the \\nContractFailA{} and \\nContractFailB{} macros are defined and that the \"two- to three-fold\" wording matches \\costRatioB--\\costRatioD.", "cost_usd": 0.12195, "latency_s": 30.5, "usage": {"input": 13467, "output": 3404}, "n_calls": 1, "model": "claude-opus-5-5", "effort": "medium", "metrics": {"n_words": 899, "preserved_rate": 0.068, "new_numbers": 0, "new_rate": 0.0, "n_author_year": 0, "n_unverifiable_citations": 0, "unverifiable_examples": [], "unknown_cite_keys": 0, "n_unsupported_claims": 0, "max_overlap_share": 0.0, "todo_placeholders": 0}, "retrieved": [], "profile": null}3{"ts": "2026-11-03T23:51:17+00:00", "task": "draft_introduction", "doc": "wp29", "domain": "out_of_domain", "condition": "A_baseline", "rep": 1, "paper": "/Users/simon-pierreboucher/Desktop/UQO/Recherche/Recherche-UQO/methodes/autonomous-research-claude-code", "degraded_literature": "", "ok": true, "text": "\\label{sec:intro}\n\nResearchers who hand scientific work to a coding agent write instructions for it, and those instructions vary widely. Some are a single sentence. Others run to several thousand words of phases, gates, ledgers, stopping rules and integrity rules, and read much like a preregistration \\citep{nosek2018preregistration}. Two literatures surround this practice.\n\n\\begin{itemize}\n \\item \\textbf{System papers.} End-to-end research systems built on large language models report manuscripts produced at low cost and reviewer scores near acceptance thresholds \\citep{lu2024aiscientist,yamada2025aiscientistv2,schmidgall2025agentlab,ifargan2025datatopaper}.\n \\item \\textbf{Audits.} Independent audits of such systems document data leakage, metric misuse, post-hoc selection and overstated novelty that are visible in execution traces but not in the papers \\citep{luo2025pitfalls,beel2025sakana}. They also find that artefacts allowing a reader to verify claims are rare \\citep{ding2026verificationgap}.\n\\end{itemize}\n\nNeither literature answers the question a delegating researcher faces: does it matter \\emph{how} the delegation is written? Each system is evaluated as a single pipeline without a control arm. The instruction regime is therefore confounded with the model, the scaffold and the amount of human co-piloting. Whether protocol structure changes what the agent delivers, and at what cost, has not been measured with the agent held fixed.\n\nThis paper measures it for one agent. The design has four elements.\n\n\\begin{itemize}\n \\item \\textbf{Runs and tasks.} We ran \\nMainRuns{} live Claude Code runs (\\nMainSessions{} sessions; client 2.1.314, model \\texttt{claude-opus-5-5}) on \\nConfTasks{} bounded scientific tasks. The tasks fall into six families: data audit; estimation and inference; forecasting under information constraints; source-grounded synthesis; theory and numerical validation; and integrated mini-projects. Each task received \\nRepeats{} fresh sessions under each of four regimes.\n \\item \\textbf{Regimes.}\n \\begin{itemize}\n \\item Arm~A: an open instruction.\n \\item Arm~B: a structured research protocol with phases, ledgers, stopping rules and recovery rules.\n \\item Arm~C: the protocol plus a controller-enforced executable gate. The gate checks the deliverable contract and reproducibility without access to the answers and returns failures to the session.\n \\item Arm~D: the same plus a separate review-and-repair session.\n \\end{itemize}\n \\item \\textbf{Execution and scoring.} Executors ran in bare mode on isolated hosts that held no copy of the reference answers. Every run was scored on a separate host by deterministic, task-specific hidden oracles, which had been validated beforehand on seeded correct and incorrect artefacts.\n \\item \\textbf{Endpoint and analysis.} The primary endpoint is intervention-free verified completion within a shared resource envelope. The primary contrast is D$-$A, with task-cluster bootstrap intervals and sign-flip permutation tests. Design, tasks and analysis were frozen with hashes before the confirmatory runs.\n\\end{itemize}\n\nThe answer is a precise null. Completion was $\\pA$ under the open instruction and $\\pD$ under the full method. The paired task-level difference is $\\dDA$, with a 95\\% interval of $[\\dDAlo, \\dDAhi]$: roughly three percentage points against the full method to six in its favour. The mechanism contrasts (B$-$A, C$-$B, D$-$C) are equally flat, and critical integrity violations were rare in every arm. Structure did buy two things: compliance with the deliverable contract, and a two- to three-fold increase in inference cost per run. No arm~C or arm~D deliverable failed the post-hoc contract check, whereas \\nContractFailA{} arm-A and \\nContractFailB{} arm-B deliverables did.\n\nTwo qualifications bound this comparison.\n\n\\begin{itemize}\n \\item Arm~D's review-and-repair loop, though available, was never triggered.\n \\item A launcher defect truncated continuation prompts (amendment A-006). The frozen comparison is therefore essentially one of protocol texts delivered through the system prompt.\n\\end{itemize}\n\nFurther evidence comes from a length-matched unstructured control and from interruption-and-resumption, stale-input, duplicate-job and missing-fixture experiments (Section~\\ref{sec:recovery}). Three held-out end-to-end mini-projects test transfer (Section~\\ref{sec:transfer}).\n\nWe do not read the null as evidence that research protocols are useless. It shows where the difficulty of delegated research lay, for a current frontier model on bounded tasks. The difficulty was not in organising the work, which the model did competently without being told. It was in \\emph{knowing what the right answer is}.\n\nEvery non-success of the frozen scoring traces, on inspection, to a defect of our oracles rather than of the executors. Four defects account for all of them:\n\n\\begin{enumerate}\n \\item an undefined feasibility label, where sessions correctly reported a month as undefined;\n \\item a generator artefact (impossible negative training hours) that sessions correctly excluded with a documented ledger, which the oracle penalised;\n \\item a rounding tolerance that no precision rule in the task justified;\n \\item a regular-expression false positive.\n\\end{enumerate}\n\nUnder a corrected rubric every arm completes every task, and the null is unchanged. In this study, then, verification was costly not because of the agents' inference budget. The cost lay in constructing tasks with known answers, oracles the executor cannot read, and checks that do not trust an exit code. The same discipline caught harness defects that would otherwise have produced confident, wrong conclusions. Examples include an evaluator invoked on the wrong workspace and a ledger key collision that aborted a resumed session (amendment A-004). This is the ``checking the checker'' problem that benchmark authors have flagged \\citep{chen2025scienceagentbench}, met here in practice.\n\n\\paragraph{Contributions.} The paper makes four contributions.\n\n\\begin{enumerate}\n \\item[(i)] \\textbf{A formalisation and reference implementation.} We formalise delegation as a contract (Eq.~\\eqref{eq:contract}). The accompanying implementation is small and auditable: a controller, a durable state store, isolated bare-mode executors, a runtime gate without access to the answers, a separate reviewer and deterministic hidden evaluators. Every claimed invariant is enforced by code, and every limit is stated.\n \\item[(ii)] \\textbf{A benchmark.} It contains \\nConfTasks{} confirmatory and \\nDevTasks{} development tasks, with hidden oracles, seeded validation sets and provenance for \\nConfRealData{} real-data and \\nConfSourceDocs{} source-document tasks, plus three transfer mini-projects.\n \\item[(iii)] \\textbf{A controlled comparison of instruction regimes.} The executing agent, tools, inputs and budget are held fixed, and the oracles are protected from the agent. The design was preregistered and frozen before execution. To our knowledge, this is the first such comparison.\n \\item[(iv)] \\textbf{A reflexive account.} We give an instrumented account of how this paper was itself produced under a mandate of the same kind, kept separate from the confirmatory evidence.\n\\end{enumerate}\n\n\\paragraph{Roadmap.} Section~\\ref{sec:related} positions the work. Sections~\\ref{sec:framework}--\\ref{sec:evaluation} present the framework, the corpus of researcher-written mandates that informed the protocol, the implementation and the experimental design. Sections~\\ref{sec:results}--\\ref{sec:transfer} report the results. Section~\\ref{sec:reflexive} gives the reflexive account. Section~\\ref{sec:limitations} states what the study cannot establish.", "change_log": "- Opening contrasts system papers and audits, then poses the delegation question; adds \\citet{ding2026verificationgap}, \\citet{schmidgall2025agentlab}, \\citet{nosek2018preregistration} and \\citet{chen2025scienceagentbench}, all with roles described in the Related work section.\n- Design, results, interpretation, contributions and roadmap follow the structure of the existing draft, with lists replacing dense run-on paragraphs.\n- All numbers are abstract/results macros or figures stated in the material. The \"three … six percentage points\" reading of the interval comes from Main results.\n- The four oracle defects are stated as in the abstract and results. \"Twelve independent sessions\" was dropped because results report eleven of twelve for F6-conf-04.\n- The corpus section is referenced through the Sections~\\ref{sec:framework}--\\ref{sec:evaluation} range because its label is not given.", "flags": "- The statement that arm D's review-repair loop was \"never triggered\" (from the abstract) sits uneasily with RC0094, where a reviewer returned *revise*. Please reconcile the wording.\n- \"Essentially one of protocol texts delivered through the system prompt\" paraphrases the existing draft's reading of A-006. Confirm the scope of the truncation (\"every continuation prompt\").\n- Check that the claim \"every arm completes every task\" under the corrected rubric (A-005/A-007) holds once the truncated Main results passage is finalised.\n- The \"first such comparison, to our knowledge\" claim is retained from the existing draft. Verify it against Section~\\ref{sec:lit-positioning}.\n- The corpus section label is unknown. Add a direct \\ref if desired.", "cost_usd": 0.13914, "latency_s": 37.1, "usage": {"input": 13281, "output": 4301}, "n_calls": 1, "model": "claude-opus-5-5", "effort": "medium", "metrics": {"n_words": 1011, "preserved_rate": 0.068, "new_numbers": 0, "new_rate": 0.0, "n_author_year": 0, "n_unverifiable_citations": 0, "unverifiable_examples": [], "unknown_cite_keys": 0, "n_unsupported_claims": 1, "max_overlap_share": 0.0, "todo_placeholders": 0}, "retrieved": [], "profile": null}4{"ts": "2026-11-03T23:51:42+00:00", "task": "draft_introduction", "doc": "wp29", "domain": "out_of_domain", "condition": "D_profiles_retrieval", "rep": 1, "paper": "/Users/simon-pierreboucher/Desktop/UQO/Recherche/Recherche-UQO/methodes/autonomous-research-claude-code", "degraded_literature": "", "ok": true, "text": "\\label{sec:intro}\n\nAutonomous research systems built on large language models now take a project from idea or data to manuscript. Their developers report low costs per paper and automated or workshop review scores near acceptance thresholds \\citep{lu2024aiscientist,yamada2025aiscientistv2,schmidgall2025agentlab,ifargan2025datatopaper}. Independent audits of the same systems tell a different story. They find data leakage, metric misuse, post-hoc selection and novelty claims that the record does not support, and these problems are visible in execution traces but not in the finished papers \\citep{luo2025pitfalls,beel2025sakana}. They also find that artifacts which would let a reader verify claims are rare \\citep{ding2026verificationgap}.\n\nBetween these two literatures sits a question that any researcher who delegates work to a coding agent faces routinely: does it matter \\emph{how} the delegation is written? The instruction files that steer such agents range from a single sentence to several thousand words of phases, gates, ledgers and integrity rules. The longer ones read like preregistrations \\citep{nosek2018preregistration}, and they are written on the premise that an agent, like a human analyst, will otherwise wander through the garden of forking paths \\citep{gelman2014statisticalcrisis}. Whether that structure changes what the agent delivers, and at what cost, has not been measured with the agent held fixed.\n\nThis paper measures it for one agent. We ran \\nMainRuns{} live Claude Code runs (\\nMainSessions{} sessions; client 2.1.314, model \\texttt{claude-opus-5-5}) on \\nConfTasks{} bounded scientific tasks in six families:\n\\begin{itemize}\n\\item data audit;\n\\item estimation and inference;\n\\item forecasting under information constraints;\n\\item source-grounded synthesis;\n\\item theory and numerical validation;\n\\item integrated mini-projects.\n\\end{itemize}\nEach task received \\nRepeats{} fresh sessions under each of four instruction regimes:\n\\begin{itemize}\n\\item[(A)] an open instruction;\n\\item[(B)] a structured research protocol with phases, ledgers, stopping and recovery rules;\n\\item[(C)] the protocol plus an executable, controller-enforced gate that checks the deliverable contract and reproducibility and returns failures to the session;\n\\item[(D)] the same plus a separate review-and-repair session.\n\\end{itemize}\nEverything else is common to all four arms: the task statement and input bundle, the tools, the bare-mode client and the resource envelope. Executors ran on nodes that held no copy of the reference answers. A deterministic, task-specific evaluator scored every run on a separate host; each evaluator had been validated beforehand on seeded correct and incorrect artefacts. The design, tasks and analysis plan were frozen with hashes before the confirmatory runs.\n\nThe primary endpoint is intervention-free verified completion within the shared envelope. A run counts as a success only if three conditions hold together: the independent rubric is satisfied, no critical integrity violation occurred, and no human intervened. The primary contrast is D$-$A, estimated with task-cluster bootstrap intervals and sign-flip permutation tests over the \\nConfTasks{} task clusters.\n\nThe answer is a precise null. Verified completion was $\\pA$ under the open instruction and $\\pD$ under the full method. The paired task-level difference is $\\dDA$, with a 95\\% interval of $[\\dDAlo, \\dDAhi]$ (permutation $p = \\dDAp$). The mechanism contrasts B$-$A, C$-$B and D$-$C are equally flat, and critical integrity violations were rare in every arm.\n\nWhat the additional structure did change was contract compliance and cost:\n\\begin{itemize}\n\\item No arm C or D deliverable failed the post-hoc contract check, whereas \\nContractFailA{} arm-A and \\nContractFailB{} arm-B deliverables did.\n\\item Inference cost per run rose by a factor of \\costRatioB{} (B), \\costRatioC{} (C) and \\costRatioD{} (D) relative to the open instruction. On these tasks, the quality--cost frontier is flat.\n\\end{itemize}\nThe comparison is narrower than the four-arm design suggests, for two reasons. First, the controller gate passed at its first cycle in every arm C and D run. Second, only \\nReviewsRevise{} of \\nReviews{} reviews returned a \\emph{revise} verdict. The one repair continuation that did occur aborted on a harness defect and was re-executed as a fresh attempt (amendment A-004). In addition, a launcher defect truncated continuation prompts (amendment A-006). The frozen experiment is therefore, in effect, a comparison of protocol texts delivered through the system prompt.\n\nThe more informative finding concerns where the residual failures came from. We inspected each of the \\nMainNonSuccess{} non-successes of the frozen scoring, and each traced to a defect in our oracles rather than in the executors.\n\\begin{itemize}\n\\item \\emph{Undefined month (F1-conf-01).} The source publishes no price index for one month, so one required growth rate is undefined. The executor sessions computed every required value and reported the task as \\texttt{partial} for exactly that reason. The oracle demanded a \\texttt{complete} label that neither the task statement nor the key specification defines.\n\\item \\emph{Impossible values (F6-conf-04).} The data generator had produced firms with negative training hours. Eleven of twelve sessions excluded them with a documented ledger and reported full-sample sensitivity estimates. The oracle's reference had used all 1,200 rows.\n\\item \\emph{Rounding tolerance.} One failure arose from a tolerance tighter than any precision rule the task stated.\n\\item \\emph{Pattern false positive.} One failure was a regular-expression false positive that flagged a correctly reported null as an effect claim.\n\\end{itemize}\nWe did not alter the frozen scoring. Under a separately reported, rubric-corrected rescoring, every arm completes every task, and the null is unchanged.\n\nWe do not read these results as evidence that research protocols are useless. We read them as evidence about where the difficulty of delegated research lies for a current frontier model on bounded tasks. The model organised the work competently without being told how. The hard part was \\emph{knowing what the right answer is}. In this study the binding cost of verification was not the agent's inference budget. It was the construction and validation of three things: tasks with known answers, oracles the executor cannot read, and checks that do not trust an exit code. The same discipline caught two harness defects that would otherwise have produced confident, wrong conclusions. One was an evaluator invoked on the wrong workspace; the other was the ledger collision behind amendment A-004. This echoes the argument that agent evaluations must be designed jointly around accuracy, cost and the integrity of the benchmark itself \\citep{kapoor2024agentsthatmatter,chen2025scienceagentbench}.\n\nFurther evidence extends the confirmatory comparison:\n\\begin{itemize}\n\\item a length-matched unstructured instruction, which separates the effect of structure from that of length;\n\\item interruption-and-resumption experiments;\n\\item stale-input, duplicate-job and missing-fixture faults;\n\\item three held-out end-to-end mini-projects (Sections~\\ref{sec:recovery} and~\\ref{sec:transfer}).\n\\end{itemize}\n\nThe paper makes four contributions.\n\\begin{enumerate}\n\\item[(i)] It formalises delegation as a contract that fixes questions, estimands, inputs, authorised actions, phase dependencies, verification criteria, budgets, recovery rules and evidence requirements before execution. It pairs this formalisation with a small, auditable reference implementation in which every claimed invariant is enforced by code and the limits of each checker are stated.\n\\item[(ii)] It supplies a benchmark of \\nConfTasks{} confirmatory and \\nDevTasks{} development tasks, with hidden oracles, seeded validation sets and documented provenance for \\nConfRealData{} real-data and \\nConfSourceDocs{} source-document tasks, plus three transfer mini-projects.\n\\item[(iii)] It reports, to our knowledge, the first preregistered comparison of instruction regimes in which the executing agent, tools, inputs and budget are held fixed and the oracles are protected from it. Existing autonomous-research systems are single pipelines evaluated by in-system reviewers or expert reading \\citep{lu2024aiscientist,schmidgall2025agentlab}. The nearest conceptual neighbour, a typed, content-addressed research protocol, is presented as an artifact rather than tested experimentally \\citep{luo2026xscientist}.\n\\item[(iv)] It gives an instrumented account of how this paper was itself produced under a mandate of the same kind, kept separate from the confirmatory evidence.\n\\end{enumerate}\n\nSection~\\ref{sec:related} positions the study in the literatures on autonomous research systems, agent benchmarks, reproducibility, preregistration and human--automation interaction. Sections~\\ref{sec:framework}--\\ref{sec:evaluation} present the conceptual framework, the corpus of researcher-written mandates that informed the protocol, the implementation, the experimental design and the evaluators. Sections~\\ref{sec:results}--\\ref{sec:transfer} report the main results, the recovery and fault experiments, the ablations and the transfer projects. Section~\\ref{sec:reflexive} gives the reflexive account. Section~\\ref{sec:limitations} states what the study cannot establish, and Section~\\ref{sec:conclusion} concludes.", "change_log": "- Move sequence: motivation (systems vs.\\ audits) → question/gap → design preview → results preview (null, compliance, cost) → design caveat (gate/review rarely exercised) → mechanism (oracle as bottleneck) → supplementary evidence → contributions with brief positioning → roadmap.\n- Length is about 1,150 words: above the 848-word target, below the JoF median of about 1,800. Literature positioning stays brief because the paper has a 2,587-word related-work section.\n- JoF conventions not applied: there is no identification-defense paragraph, because the treatment is randomized by design and identification is not contested. There is also no economic-magnitude framing beyond the cost ratios, which are the natural magnitude here.\n- Kept the macros from the manuscript rather than numeric literals. Reported the F6-conf-04 count as \"eleven of twelve\", following the Main results section.\n- Rewrote the review-and-repair description as stated facts (A-004, A-006, \\nReviewsRevise{}/\\nReviews{}) instead of the current draft's \"never triggered\".\n- Reformatted the arm definitions, the failure groups, the supplementary evidence and the contributions as lists for readability.", "flags": "- Inconsistency: the abstract says \"a generator artefact that twelve sessions correctly excluded\", but Main results says eleven of twelve excluded and one passed. Reconcile.\n- Inconsistency: the abstract and the current intro say arm D's review-repair loop was \"never triggered\", yet RC0094 received a *revise* verdict and a continuation (A-004). Confirm the intended wording.\n- The claim that \"each of the \\nMainNonSuccess{} non-successes traced to oracle defects\" rests on the authors' inspection and on Appendix~\\ref{app:evaluators}. Verify that it holds for all of them.\n- Citation uses to confirm: \\citet{kapoor2024agentsthatmatter} and \\citet{chen2025scienceagentbench} for cost- and oracle-aware evaluation, and \\citet{nosek2018preregistration} as the preregistration analogy.\n- The \"evaluator invoked on the wrong workspace\" defect appears only in the existing intro draft. Confirm that it is documented elsewhere, for example in the amendment ledger.\n- Verify the \"first ... to our knowledge\" claim against the related-work survey.\n- The \\dDAp{} permutation $p$ is taken from Main results. The section label for corpus-informed method design was not provided, so it is covered only by the range reference.", "cost_usd": 0.17949, "latency_s": 49.6, "usage": {"input": 16388, "output": 5697}, "n_calls": 1, "model": "claude-opus-5-5", "effort": "medium", "metrics": {"n_words": 1272, "preserved_rate": 0.082, "new_numbers": 0, "new_rate": 0.0, "n_author_year": 0, "n_unverifiable_citations": 0, "unverifiable_examples": [], "unknown_cite_keys": 0, "n_unsupported_claims": 0, "max_overlap_share": 0.0, "todo_placeholders": 0}, "retrieved": [{"unit": "intro_move", "score": 0.9633, "doc_id": "jof_12766", "doi": "10.1111/jofi.12766", "title": "Labor-Technology Substitution: Implications for Asset Pricing", "year": 2019, "page": 1, "section": "introduction", "sha256": "2f9e655c4aaab4f1", "role": "introduction", "genre": "mixed_theory_empirical", "subfield": "asset_pricing", "confidence": 0.8, "split": "train"}, {"unit": "intro_move", "score": 0.8637, "doc_id": "jof_12844", "doi": "10.1111/jofi.12844", "title": "Trading Against the Random Expiration of Private Information: A Natural Experiment", "year": 2020, "page": 3, "section": "introduction", "sha256": "649cb23d513bbe54", "role": "introduction", "genre": "mixed_theory_empirical", "subfield": "market_microstructure", "confidence": 0.8, "split": "train"}, {"unit": "intro_move", "score": 0.8193, "doc_id": "jof_12544", "doi": "10.1111/jofi.12544", "title": "Why Does Return Predictability Concentrate in Bad Times?", "year": 2017, "page": 3, "section": "introduction", "sha256": "9650ee6f9da03c5e", "role": "introduction", "genre": "mixed_theory_empirical", "subfield": "asset_pricing", "confidence": 0.8, "split": "train"}, {"unit": "identification", "score": 0.8183, "doc_id": "jof_12526", "doi": "10.1111/jofi.12526", "title": "Firm Investment and Stakeholder Choices: A Top-Down Theory of Capital Budgeting", "year": 2017, "page": 1, "section": NaN, "sha256": "a9f703cc3137b189", "role": "empirical_design", "genre": "theoretical", "subfield": "corporate_finance", "confidence": 0.8, "split": "train"}, {"unit": "intro_move", "score": 0.8148, "doc_id": "jof_12846", "doi": "10.1111/jofi.12846", "title": "Access to Collateral and the Democratization of Credit: France’s Reform of the Napoleonic Security Code", "year": 2020, "page": 2, "section": "introduction", "sha256": "b5dfbca07d3c9237", "role": "introduction", "genre": "empirical", "subfield": "corporate_finance", "confidence": 0.8, "split": "train"}, {"unit": "intro_move", "score": 0.8145, "doc_id": "jof_12544", "doi": "10.1111/jofi.12544", "title": "Why Does Return Predictability Concentrate in Bad Times?", "year": 2017, "page": 3, "section": "introduction", "sha256": "9650ee6f9da03c5e", "role": "introduction", "genre": "mixed_theory_empirical", "subfield": "asset_pricing", "confidence": 0.8, "split": "train"}], "profile": "empirical_all"}5{"ts": "2026-11-03T23:51:49+00:00", "task": "improve_abstract", "doc": "wp29", "domain": "out_of_domain", "condition": "B_strong_prompt", "rep": 1, "paper": "/Users/simon-pierreboucher/Desktop/UQO/Recherche/Recherche-UQO/methodes/autonomous-research-claude-code", "degraded_literature": "", "ok": true, "text": "\\begin{abstract}\nInstruction files that steer coding agents range from one sentence to thousands of words of phases, gates and integrity rules. Does that structure change what the agent delivers? We provide, to our knowledge, the first controlled test that holds the agent fixed. We ran \\nMainRuns{} live Claude Code runs on \\nConfTasks{} bounded tasks, with \\nRepeats{} sessions per task under each of four regimes: an open instruction; a structured protocol; the protocol plus an executable gate; and the gated protocol plus a separate reviewer. Hidden oracles scored every run, and the design was frozen beforehand. Intervention-free verified completion was \\pA{} under the open instruction and \\pD{} under the full method (paired difference \\dDA{}, 95\\% interval [\\dDAlo, \\dDAhi]), and the mechanism contrasts are equally null. Structure bought full contract compliance at two to three times the inference cost per run. Every non-success was, on inspection, an oracle defect. For bounded tasks with known answers, structure buys auditability rather than correctness, and the scarce resource is a trustworthy oracle.\n\\end{abstract}", "change_log": "- Cut the abstract to about 150 words, at the upper bound of the brief's 100–150 range, from about 370.\n- Opened with the motivating fact and stated the question directly as a question.\n- Made the contribution explicit. The \"first controlled test, to our knowledge\" wording is taken from contribution (iii) in the Introduction.\n- Kept every magnitude in the original wording and macros: \\pA{}, \\pD{}, \\dDA{}, [\\dDAlo, \\dDAhi], \"two to three times\", \"full compliance\".\n- Repaired \"95\\ interval\" in the source abstract to \"95\\% interval\".\n- Dropped the following detail to meet the word limit. It remains in the body.\n - Client version and model name.\n - The six task families.\n - Bare-mode and isolated-host details.\n - Seeded oracle validation and hash freezing, now compressed to \"frozen beforehand\".\n - The four oracle-defect examples.\n - The corrected-rubric result.\n - The fact that arm D's review loop was never triggered.\n - The recovery/fault experiments, length-matched control and mini-projects.\n - The reflexive account and the rarity of integrity violations.\n- Kept the JEL codes and keywords in metadata. They are not repeated in the text.", "flags": "- **Omitted claims.** The brief's word limit forced me to leave out claims that the original abstract stated. Please confirm these omissions are acceptable or name one or two to restore, which would cost about 15–25 words each:\n - \"under a corrected rubric every arm completes every task, and the null is unchanged\".\n - \"Arm D's review-repair loop … was never triggered\".\n - \"critical integrity violations were rare in every arm\".\n - The robustness experiments and the reflexive account.\n- **Arm D framing.** The Introduction notes that a launcher defect (amendment A-006) truncated continuation prompts. As a result, the frozen comparison is effectively one of protocol texts delivered via the system prompt. The abstract labels arm D a regime with \"a separate reviewer\". Consider whether a brief qualifier is needed for proportionality.\n- **\"Every non-success was, on inspection, an oracle defect.\"** This relies on the post-hoc inspection and the evaluator audit (Appendix~\\ref{app:evaluators}). The wording is retained from the original, but please confirm it holds for all \\nMainNonSuccess{} cases.\n- **Macro values.** Please verify that \\pA{}, \\pD{}, \\dDA{}, \\dDAlo and \\dDAhi resolve to the intended values. I have not seen their numeric expansions.", "cost_usd": 0.14726, "latency_s": 40.5, "usage": {"input": 13230, "output": 4717}, "n_calls": 1, "model": "claude-opus-5-5", "effort": "medium", "metrics": {"n_words": 168, "preserved_rate": 0.014, "new_numbers": 0, "new_rate": 0.0, "n_author_year": 0, "n_unverifiable_citations": 0, "unverifiable_examples": [], "unknown_cite_keys": 0, "n_unsupported_claims": 0, "max_overlap_share": 0.0, "todo_placeholders": 0}, "retrieved": [], "profile": null}6{"ts": "2026-11-03T23:51:49+00:00", "task": "improve_abstract", "doc": "wp29", "domain": "out_of_domain", "condition": "C_conventional_rag", "rep": 1, "paper": "/Users/simon-pierreboucher/Desktop/UQO/Recherche/Recherche-UQO/methodes/autonomous-research-claude-code", "degraded_literature": "", "ok": true, "text": "\\begin{abstract}\nInstruction files for research coding agents range from one sentence to several thousand words of phases, gates and integrity rules. Does that structure change what the agent delivers? We answer with a preregistered experiment that holds the agent fixed: \\nMainRuns{} live Claude Code runs (client 2.1.314, model \\texttt{claude-opus-5-5}) on \\nConfTasks{} bounded scientific tasks, \\nRepeats{} fresh sessions per task under each of four regimes, from an open instruction to a protocol with an executable gate and separate review. Hidden deterministic oracles scored every run. Intervention-free verified completion was \\pA{} under the open instruction and \\pD{} under the full method (paired difference \\dDA{}; 95\\% interval [\\dDAlo, \\dDAhi]). Structure bought full contract compliance at two to three times the inference cost per run. On inspection, every non-success was an oracle defect. For bounded tasks with known answers, structure buys auditability, not correctness, and the scarce resource is a trustworthy oracle.\n\\end{abstract}", "change_log": "- **Length.** The abstract was cut from about 330 words to about 147 to meet the 100–150-word target in the brief.\n- **Order.** It now runs: motivation, then an explicit question, then the design framed as the contribution (\"preregistered experiment that holds the agent fixed\"), then the headline magnitudes, then the cost result, then the oracle finding, then the implication.\n- **Values kept unchanged.** All retained macros and values are unchanged: \\nMainRuns, client 2.1.314, model, \\nConfTasks, \\nRepeats, four regimes, \\pA, \\pD, \\dDA, [\\dDAlo, \\dDAhi], and \"two to three times the inference cost per run.\"\n- **Typo fixed.** \"95\\ interval\" was a typo; it now reads \"95\\% interval.\"\n- **\"Preregistered.\"** This wording is taken from contribution (iii) in the Introduction (\"preregistered and frozen with hashes\").\n- **Phrasing tightened.** \"On inspection, every non-success was an oracle defect\" is a shortened version of the original \"defect of our oracles rather than of the executors.\"\n- **Omitted for length.** The following were dropped, and none was altered:\n - \\nMainSessions{} and the six families.\n - The bare-mode and isolated-host details, seeded oracle validation, and the hash freeze.\n - The null mechanism contrasts and the rarity of critical integrity violations.\n - The four specific oracle defects, the corrected-rubric result, and the fact that the D review-repair loop was never triggered.\n - The robustness and fault experiments, the transfer mini-projects, and the reflexive production account.", "flags": "- **Corrected-rubric result.** The abstract does not state that \"under a corrected rubric every arm completes every task, and the null is unchanged.\" Consider reinstating it, or swapping it for the cost sentence, if you can spare about 12 words.\n- **Never-triggered review loop.** Dropping the fact that the D review-repair loop was never triggered, and the related launcher truncation (A-006), may let readers overstate what arm D tested. Consider whether a word-limit exception is warranted.\n- **Wording of the regimes.** Check that \"from an open instruction to a protocol with executable gate and separate review\" adequately conveys the four regimes (A–D) for your readership.\n- **Word count with macros.** The count assumes each macro renders as a single token. Verify the final count after compilation.", "cost_usd": 0.14079, "latency_s": 36.2, "usage": {"input": 14488, "output": 4142}, "n_calls": 1, "model": "claude-opus-5-5", "effort": "medium", "metrics": {"n_words": 149, "preserved_rate": 0.041, "new_numbers": 0, "new_rate": 0.0, "n_author_year": 0, "n_unverifiable_citations": 0, "unverifiable_examples": [], "unknown_cite_keys": 0, "n_unsupported_claims": 0, "max_overlap_share": 0.0, "todo_placeholders": 0}, "retrieved": [{"unit": "section_opening", "score": 0.9083, "doc_id": "jof_12766", "doi": "10.1111/jofi.12766", "title": "Labor-Technology Substitution: Implications for Asset Pricing", "year": 2019, "page": 1, "section": "Introduction (unnumbered)", "sha256": "2f9e655c4aaab4f1", "role": "introduction", "genre": "mixed_theory_empirical", "subfield": "asset_pricing", "confidence": 0.963, "split": "train"}, {"unit": "section_opening", "score": 0.8655, "doc_id": "jof_12722", "doi": "10.1111/jofi.12722", "title": "Collateral Constraints and the Law of One Price: An Experiment", "year": 2018, "page": 4, "section": "Theory", "sha256": "ab134dba0b0dcf60", "role": "theory", "genre": "mixed_theory_empirical", "subfield": "asset_pricing", "confidence": 0.963, "split": "train"}, {"unit": "section_opening", "score": 0.8515, "doc_id": "jof_12530", "doi": "10.1111/jofi.12530", "title": "Presidential Address: The Scientific Outlook in Financial Economics", "year": 2017, "page": 36, "section": "Recommendations", "sha256": "e7091a0d4b8826fc", "role": "other", "genre": "survey_or_address", "subfield": "financial_econometrics", "confidence": 0.938, "split": "train"}, {"unit": "table_notes", "score": 0.8278, "doc_id": "jof_12619", "doi": "10.1111/jofi.12619", "title": "What Makes a Good Trader? On the Role of Intuition and Reflection on Trader Performance", "year": 2018, "page": 8, "section": "Table I", "sha256": "034e718ad966ca13", "role": "results", "genre": "empirical", "subfield": "behavioral_finance", "confidence": 0.9, "split": "train"}, {"unit": "section_opening", "score": 0.8266, "doc_id": "jof_12722", "doi": "10.1111/jofi.12722", "title": "Collateral Constraints and the Law of One Price: An Experiment", "year": 2018, "page": 28, "section": "Conclusion", "sha256": "ab134dba0b0dcf60", "role": "conclusion", "genre": "mixed_theory_empirical", "subfield": "asset_pricing", "confidence": 0.963, "split": "train"}, {"unit": "section_opening", "score": 0.8227, "doc_id": "jof_12619", "doi": "10.1111/jofi.12619", "title": "What Makes a Good Trader? On the Role of Intuition and Reflection on Trader Performance", "year": 2018, "page": 6, "section": "Design", "sha256": "034e718ad966ca13", "role": "empirical_design", "genre": "empirical", "subfield": "behavioral_finance", "confidence": 0.938, "split": "train"}], "profile": null}7{"ts": "2026-11-03T23:51:52+00:00", "task": "improve_abstract", "doc": "wp29", "domain": "out_of_domain", "condition": "A_baseline", "rep": 1, "paper": "/Users/simon-pierreboucher/Desktop/UQO/Recherche/Recherche-UQO/methodes/autonomous-research-claude-code", "degraded_literature": "", "ok": true, "text": "\\noindent Coding-agent instruction files range from one sentence to several thousand words of phases, gates and integrity rules. With the agent fixed, does that structure change what it delivers? We ran \\nMainRuns{} Claude Code runs (client 2.1.314, model \\texttt{claude-opus-5-5}) on \\nConfTasks{} bounded tasks, \\nRepeats{} fresh sessions per task under four regimes: open instruction; structured protocol; protocol plus executable gate; and the same plus separate review-and-repair. Hidden oracles scored every run under a prefrozen design. Intervention-free verified completion was \\pA{} under the open instruction and \\pD{} under the full method (paired difference \\dDA{}, 95\\% interval [\\dDAlo, \\dDAhi]); mechanism contrasts are equally null. Structure bought full deliverable-contract compliance at two to three times the inference cost per run. On inspection, every non-success was an oracle defect; under a corrected rubric every arm completes every task. For bounded tasks with known answers, structure buys auditability, not correctness; the scarce resource is a trustworthy oracle.", "change_log": "- Restructured the abstract as question, design, result, cost, diagnosis and implication. The question is now an explicit sentence, and the contribution is stated in the closing sentence.\n- Compressed the text to about 150 words, per the brief. The brief's 100–150-word limit conflicts with the instruction to preserve every item of the original (~330-word) abstract. To meet the limit, I dropped secondary details rather than altering any of them:\n - the session count (\\nMainSessions{}) and the six task families;\n - bare mode, isolated hosts, oracle validation on seeded artefacts, the separate scoring host, and hashing;\n - the \"task-cluster bootstrap\" label;\n - the rarity of critical integrity violations;\n - the four specific oracle defects;\n - the fact that Arm D's review-repair loop was never triggered;\n - the fault, length-matched and transfer experiments;\n - the reflexive production account.\n- Every number and macro that is retained is unchanged. I added no citations, because abstracts conventionally carry none and the original had none.", "flags": "- Please confirm which omitted items you want restored. I would rank them in this order:\n - Arm D's review-repair loop was never triggered. The Introduction notes that, because of this and the A-006 launcher defect, the frozen comparison is effectively one of protocol texts delivered through the system prompt. Dropping this arguably weakens how the D$-$A contrast reads.\n - The rarity of critical integrity violations.\n - The \\nMainSessions{} count.\n- \"Hidden oracles scored every run under a prefrozen design\" condenses \"design, tasks and analysis were frozen with hashes before the confirmatory runs.\" Please verify that this paraphrase is acceptable.\n- \"Structure bought full deliverable-contract compliance\" attributes compliance to the protocol, gate and reviewer jointly, as the original does. Note that the Introduction reports contract failures in arms A and B (\\nContractFailA{}, \\nContractFailB{}) but none in arms C or D. Strictly, then, full compliance holds for arms C and D, not for arm B alone. Consider \"the gate (arms C–D) bought…\" if you want precision.\n- The original abstract contains \"95\\ interval\" (a missing \\%). This version uses \"95\\%\", consistent with the Introduction.", "cost_usd": 0.16682, "latency_s": 50.9, "usage": {"input": 13044, "output": 5732}, "n_calls": 1, "model": "claude-opus-5-5", "effort": "medium", "metrics": {"n_words": 151, "preserved_rate": 0.041, "new_numbers": 0, "new_rate": 0.0, "n_author_year": 0, "n_unverifiable_citations": 0, "unverifiable_examples": [], "unknown_cite_keys": 0, "n_unsupported_claims": 0, "max_overlap_share": 0.0, "todo_placeholders": 0}, "retrieved": [], "profile": null}8{"ts": "2026-11-03T23:52:01+00:00", "task": "improve_abstract", "doc": "wp29", "domain": "out_of_domain", "condition": "D_profiles_retrieval", "rep": 1, "paper": "/Users/simon-pierreboucher/Desktop/UQO/Recherche/Recherche-UQO/methodes/autonomous-research-claude-code", "degraded_literature": "", "ok": true, "text": "Does how research is delegated to a coding agent change what it delivers? Holding the agent fixed (Claude Code 2.1.314, \\texttt{claude-opus-5-5}), we ran \\nMainRuns{} live runs on \\nConfTasks{} bounded tasks in six families, \\nRepeats{} fresh sessions per task under four regimes frozen before execution: an open instruction; a structured protocol; the protocol plus an executable gate; or these plus a separate review-and-repair session. Oracles hidden from the executors scored every run. Intervention-free verified completion was \\pA{} under the open instruction and \\pD{} under the full method; the paired task-level difference is \\dDA{} (task-cluster bootstrap 95\\% interval [\\dDAlo, \\dDAhi]), and the mechanism contrasts are equally null. Structure bought full compliance with the deliverable contract at two to three times the inference cost per run. Every non-success, on inspection, was an oracle defect. For bounded tasks with known answers, structure buys auditability, not correctness, and the scarce resource is a trustworthy oracle.", "change_log": "- **Move sequence:** question → design (agent held fixed, four regimes, protected oracles) → primary magnitude with interval → mechanism null → what structure buys and its cost → diagnosis of failures → bounded implication.\n- **Corpus conventions applied:**\n - Question opener.\n - Method and magnitude moves.\n - Bounded closing claim.\n- **Corpus conventions not applied:**\n - No literature-gap sentence. It does not fit the word budget, and the gap is already implicit in \"holding the agent fixed.\"\n - No sentence-count target, because this is a methods/evaluation paper and not an empirical-finance paper.\n- **Length:** The text is about 150 words. Brief-mandated limit is 100–150, versus about 330 in the source abstract.\n- **Content omitted to meet the limit (not altered):**\n - \\nMainSessions{} sessions.\n - Bare mode and isolated hosts.\n - Seeded oracle validation.\n - Hash freezing (condensed to \"frozen before execution\").\n - Rare critical violations in every arm.\n - The four specific oracle defects.\n - The corrected rubric under which every arm completes every task and the null is unchanged.\n - The never-triggered review-repair loop.\n - Fault, interruption and length-matched experiments.\n - The transfer mini-projects.\n - The reflexive production account.\n- **Content compressed:** \"the protocol, gate and reviewer\" is shortened to \"structure.\" \"a defect of our oracles rather than of the executors\" is shortened to \"an oracle defect.\"\n- **Typo fixed:** The source text \"95\\ interval\" became \"95\\% interval,\" matching the introduction.\n- **Macros:** All numeric macros are kept exactly as in the manuscript.", "flags": "- **Omitted findings:** Check that it is acceptable to drop the following from the abstract:\n - \"critical integrity violations were rare in every arm\"\n - \"under a corrected rubric every arm completes every task, and the null is unchanged\"\n - \"Arm D's review-repair loop … was never triggered\"\n\n The last two are material qualifiers of the null. Consider relaxing the 150-word cap to retain them.\n- **Amendment A-006 not mentioned:** The introduction notes that a launcher defect truncated every continuation prompt (amendment A-006), so the comparison is effectively one of protocol texts delivered via the system prompt. Decide whether the abstract needs this caveat.\n- **\"Frozen before execution\":** This paraphrases \"frozen with hashes before the confirmatory runs.\" Check that it is acceptable.\n- **\"Every non-success … was an oracle defect\":** The executor-side half of the original contrast (\"rather than of the executors\") is now implicit. It is supported by the Main results section, but confirm the shortened wording is acceptable.\n- **Unresolved macros:** The macros \\pA, \\pD, \\dDA, \\dDAlo, \\dDAhi, \\nMainRuns, \\nConfTasks and \\nRepeats must resolve in the compiled document.", "cost_usd": 0.17096, "latency_s": 47.2, "usage": {"input": 16151, "output": 5318}, "n_calls": 1, "model": "claude-opus-5-5", "effort": "medium", "metrics": {"n_words": 150, "preserved_rate": 0.041, "new_numbers": 0, "new_rate": 0.0, "n_author_year": 0, "n_unverifiable_citations": 0, "unverifiable_examples": [], "unknown_cite_keys": 0, "n_unsupported_claims": 0, "max_overlap_share": 0.0, "todo_placeholders": 0}, "retrieved": [{"unit": "intro_move", "score": 0.9633, "doc_id": "jof_12766", "doi": "10.1111/jofi.12766", "title": "Labor-Technology Substitution: Implications for Asset Pricing", "year": 2019, "page": 1, "section": "introduction", "sha256": "2f9e655c4aaab4f1", "role": "introduction", "genre": "mixed_theory_empirical", "subfield": "asset_pricing", "confidence": 0.8, "split": "train"}, {"unit": "intro_move", "score": 0.8637, "doc_id": "jof_12844", "doi": "10.1111/jofi.12844", "title": "Trading Against the Random Expiration of Private Information: A Natural Experiment", "year": 2020, "page": 3, "section": "introduction", "sha256": "649cb23d513bbe54", "role": "introduction", "genre": "mixed_theory_empirical", "subfield": "market_microstructure", "confidence": 0.8, "split": "train"}, {"unit": "intro_move", "score": 0.8193, "doc_id": "jof_12544", "doi": "10.1111/jofi.12544", "title": "Why Does Return Predictability Concentrate in Bad Times?", "year": 2017, "page": 3, "section": "introduction", "sha256": "9650ee6f9da03c5e", "role": "introduction", "genre": "mixed_theory_empirical", "subfield": "asset_pricing", "confidence": 0.8, "split": "train"}, {"unit": "identification", "score": 0.8183, "doc_id": "jof_12526", "doi": "10.1111/jofi.12526", "title": "Firm Investment and Stakeholder Choices: A Top-Down Theory of Capital Budgeting", "year": 2017, "page": 1, "section": NaN, "sha256": "a9f703cc3137b189", "role": "empirical_design", "genre": "theoretical", "subfield": "corporate_finance", "confidence": 0.8, "split": "train"}, {"unit": "intro_move", "score": 0.8148, "doc_id": "jof_12846", "doi": "10.1111/jofi.12846", "title": "Access to Collateral and the Democratization of Credit: France’s Reform of the Napoleonic Security Code", "year": 2020, "page": 2, "section": "introduction", "sha256": "b5dfbca07d3c9237", "role": "introduction", "genre": "empirical", "subfield": "corporate_finance", "confidence": 0.8, "split": "train"}, {"unit": "intro_move", "score": 0.8145, "doc_id": "jof_12544", "doi": "10.1111/jofi.12544", "title": "Why Does Return Predictability Concentrate in Bad Times?", "year": 2017, "page": 3, "section": "introduction", "sha256": "9650ee6f9da03c5e", "role": "introduction", "genre": "mixed_theory_empirical", "subfield": "asset_pricing", "confidence": 0.8, "split": "train"}], "profile": "empirical_all"}9{"ts": "2026-11-03T23:52:20+00:00", "task": "draft_introduction", "doc": "wp29", "domain": "out_of_domain", "condition": "E_full_engine", "rep": 1, "paper": "/Users/simon-pierreboucher/Desktop/UQO/Recherche/Recherche-UQO/methodes/autonomous-research-claude-code", "degraded_literature": "", "ok": true, "text": "\\label{sec:intro}\n\nThe instruction files that steer coding agents through research tasks range from a single sentence to several thousand words of phases, gates, ledgers and integrity rules, and the longer ones read much like preregistrations \\citep{nosek2018preregistration}. They rest on a plausible premise. The analytic freedom that produces false positives in human research \\citep{simmons2011falsepositive,gelman2014statisticalcrisis} does not disappear when the analyst is a language model, so written structure should discipline it. Whether that premise holds has not been measured with the agent held fixed: we do not know whether the structure of the instruction changes what the agent delivers, or what it costs. This paper provides that measurement for one agent.\n\nThe question falls between two literatures that do not address it directly. End-to-end research systems report reviewable manuscripts at low cost \\citep{lu2024aiscientist,yamada2025aiscientistv2,schmidgall2025agentlab,ifargan2025datatopaper}. Independent audits document data leakage, metric misuse, post-hoc selection and wrongly claimed novelty that are visible in execution traces but not in the papers \\citep{luo2025pitfalls,beel2025sakana}. A survey of the field finds reproducibility-grade and claim-verification artefacts rare \\citep{ding2026verificationgap}. Each system, however, is a single pipeline without a control arm, judged by an in-system reviewer or by expert reading. Task-level benchmarks supply executable oracles but hold the instruction fixed rather than varying it \\citep{chen2025scienceagentbench}. Protocol designs that type and gate every research object \\citep{luo2026xscientist} are artefacts, not experiments.\n\nWe vary only the instruction. We ran \\nMainRuns{} live Claude Code runs (\\nMainSessions{} sessions; client 2.1.314, model \\texttt{claude-opus-5-5}) on \\nConfTasks{} bounded scientific tasks in six families: data audit, estimation and inference, forecasting under information constraints, source-grounded synthesis, theory and numerical validation, and integrated mini-projects. Each task received \\nRepeats{} fresh sessions under each of four regimes. Arm A used an open instruction. Arm B used a structured protocol with phases, ledgers, stopping and recovery rules. Arm C added an executable, controller-enforced gate that checks the deliverable contract and reproducibility and returns failures to the session. Arm D added a separate review-and-repair session. Tools, inputs and the resource envelope were identical across arms. Executors ran in bare mode on nodes that held no copy of the reference answers. Deterministic, task-specific oracles scored every run on a separate host, and each oracle was validated beforehand on seeded correct and incorrect artefacts. The design, tasks and analysis were frozen with hashes before the confirmatory runs.\n\nThe primary endpoint is intervention-free verified completion within the shared envelope. This endpoint separates workflow autonomy from scientific correctness (Section~\\ref{sec:framework}). The primary contrast is D$-$A, paired within task, with task-cluster bootstrap intervals and sign-flip permutation tests over the \\nConfTasks{} tasks. The estimand is therefore the effect of the instruction regime for this model and client version on this task population. Extrapolating it to harder or open-ended tasks, or to other agents, requires assumptions that the design does not test.\n\nThe answer is a null. Verified completion was $\\pA$ under the open instruction and $\\pD$ under the full method. The paired difference is $\\dDA$ (95\\% interval $[\\dDAlo, \\dDAhi]$), which spans roughly three percentage points against D to roughly six in its favour. The mechanism contrasts B$-$A, C$-$B and D$-$C are equally flat, and critical integrity violations were rare in every arm. Because completion under the open instruction was already high, the design has limited room to detect improvements. The null therefore speaks to bounded tasks at this difficulty, not to protocols in general.\n\nWhere the added structure did make a difference was compliance with the deliverable contract. No arm C or D deliverable failed the post-hoc contract check, against \\nContractFailA{} in arm A and \\nContractFailB{} in arm B. The price was two to three times the inference cost per run, measured at client list prices.\n\nTwo features of the implementation qualify the contrast. First, arm D's review-and-repair branch, though available, was never triggered in the benchmark. Second, a launcher defect truncated every continuation prompt (amendment A-006). The frozen comparison is therefore in effect one of protocol texts delivered through the system prompt. We keep the frozen scoring as the confirmatory result and report separate post-fix verification runs that are not pooled with it.\n\nThe more informative finding concerns where the failures lie. On inspection, every non-success of the frozen scoring traces to a defect of our oracles rather than of the executors. On one task the source publishes no consumer-price index for one window month. All twelve sessions reported the resulting growth rate as undefined, and all were scored as failures because the oracle required a feasibility label that the task never defined. On another task the data generator had produced impossible negative training hours. Eleven of twelve sessions excluded those rows with a documented ledger, against an oracle that kept them. The remaining failures were a rounding tolerance that no precision rule justified and a regular-expression false positive, the latter being the only arm-specific critical violation. Under a corrected rubric every arm completes every task, and the null is unchanged.\n\nThese cases suggest that, for this model on these tasks, the difficulty of delegated research lay less in organising the work than in specifying and checking the right answer. Harness-level defects point the same way. An evaluator was invoked on the wrong workspace, and a ledger key collision aborted one run (amendment A-004). Both would have yielded confident and wrong conclusions had the checks trusted exit codes. Because the same organisation wrote the tasks, oracles and treatments, these diagnoses are our own and are documented for external audit (Section~\\ref{sec:limitations}).\n\nThe paper makes four contributions. First, it formalises delegation as a contract. It also provides a small reference implementation (controller, durable state, isolated bare-mode executors, an oracle-free runtime gate, separate review and hidden deterministic evaluators) in which every claimed invariant is enforced in code (Sections~\\ref{sec:framework} and~\\ref{sec:implementation}). Second, it contributes a benchmark of \\nConfTasks{} confirmatory and \\nDevTasks{} development tasks with hidden oracles, seeded validation sets and provenance for \\nConfRealData{} real-data and \\nConfSourceDocs{} source-document tasks, together with three held-out transfer mini-projects. Third, it reports what is, to our knowledge, the first comparison of instruction regimes frozen before execution, with the executing agent, tools, inputs and budget held fixed and the oracles protected from it. A length-matched unstructured control and the interruption-and-resumption, stale-input, duplicate-job and missing-fixture experiments complement it. Fourth, it gives an instrumented account of how this paper was produced under a mandate of the same kind, kept separate from the confirmatory evidence.\n\nFor the kind of task a laboratory would first consider delegating, the marginal value of protocol structure lies in auditability rather than correctness, and the scarce input is a trustworthy oracle. Section~\\ref{sec:related} positions the work. Sections~\\ref{sec:framework}--\\ref{sec:evaluation} present the framework, the corpus of researcher-written mandates that informed the method, the implementation and the design. Sections~\\ref{sec:results}--\\ref{sec:transfer} report the results. Section~\\ref{sec:reflexive} gives the reflexive account, and Section~\\ref{sec:limitations} states what the study cannot establish.", "change_log": "- Move sequence: motivation (instruction files and analytic freedom) → literature gap (systems vs. audits vs. benchmarks) → preview of approach (design, in an itemised list) → preview of results (null plus compliance and cost) → identification caveats (A-006, untriggered reviewer) → interpretation (oracle defects as the substantive finding, in an itemised list) → contributions → bounded implication and roadmap.\n- The introduction opens with the researcher's practical problem rather than the system literature. This puts the question in the first paragraph, as JoF-style introductions do.\n- Length is about 1,050 words, above the 848-word allotment but well below the JoF median of about 1,800. A methods/CS paper with a separate 2,587-word related-work section does not need a long in-intro literature review.\n- Some JoF conventions were not applied: there are no stars or t-statistics, and economic-significance framing is replaced by cost ratios, since the evidence is interval-based. Statistical language is limited to the bootstrap interval, as in the manuscript.\n- The contract-compliance counts, the A-004/A-006 amendments and the oracle-defect examples are kept because they appear in the current draft and in the results section.\n- The \"twelve sessions\" statement was split into \"all twelve\" (price-index task) and \"eleven of twelve\" (training-hours task) to match the results section.\n--- referee pass ---\n- **Audit:** No new numbers and no unknown keys were found. The \"missing\" numbers belong to other sections and are not needed here. All macros and amendment IDs are kept verbatim.\n- **Opening:** Removed the unsupported trend claim (\"Researchers increasingly hand…\"). The paragraph now opens with the observation the abstract supports.\n- **Null interval:** Replaced \"exclude effects larger than a few percentage points in either direction\" with the asymmetric reading stated in Main results (about three points against D to about six for D). The original phrasing misdescribed the interval.\n- **Identification and scope:** Added a paragraph on identification. It states the paired within-task design, inference at the level of task clusters, an estimand conditional on model, client and task population, and the ceiling effect noted in the Conclusion. These bound what the null supports.\n- **Post-fix runs:** Corrected \"We report both rather than re-run\". The material contains post-fix verification runs (tab:postfix), not pooled with the confirmatory results.\n- **Contract and cost:** Labelled the contract check as post-hoc and the costs as list-price estimates, matching the table captions.\n- **Benchmark wording:** Softened the benchmark characterisation to \"hold the instruction fixed\", which is directly supported.\n- **Interpretive claim:** Hedged \"the model did competently without being told\" to \"for this model on these tasks\".\n- **Authorship caveat:** Added the self-authorship caveat from the Conclusion next to the oracle-defect diagnoses.\n- **Format:** Converted the bullet lists to prose, which is the norm for journal introductions. Length is trimmed toward the target.", "flags": "- **Arm D trigger:** \"Review-and-repair never triggered\" (abstract) sits alongside Main results, where run RC0094's reviewer returned *revise* and \\nReviewsRevise{} reviews returned *revise*. Please reconcile, for example by stating \"never triggered in any completed confirmatory attempt\".\n- **Workspace defect:** The evaluator-on-wrong-workspace defect has no amendment ID in the material provided. Please add one if it exists.\n- **\"Preregistered\":** The draft's \"preregistered\" was replaced with \"frozen before execution\", because the material documents only hash-freezing. Restore it only if a registry entry exists, and if so cite it.\n- **Ceiling interpretation:** Confirm that $\\pA$ is high enough that \"limited room to detect improvements\" is accurate. The Conclusion says the ceiling was high, but the macro values are not visible here.\n- **Contract check by construction:** Arms C and D's gate enforces the deliverable contract. Consider stating explicitly that their perfect compliance is partly by construction.", "cost_usd": 0.32482, "latency_s": 85.4, "usage": {"input": 32814, "output": 9678}, "n_calls": 2, "model": "claude-opus-5-5", "effort": "medium", "metrics": {"n_words": 1118, "preserved_rate": 0.068, "new_numbers": 0, "new_rate": 0.0, "n_author_year": 0, "n_unverifiable_citations": 0, "unverifiable_examples": [], "unknown_cite_keys": 0, "n_unsupported_claims": 0, "max_overlap_share": 0.0, "todo_placeholders": 0}, "retrieved": [{"unit": "intro_move", "score": 0.9633, "doc_id": "jof_12766", "doi": "10.1111/jofi.12766", "title": "Labor-Technology Substitution: Implications for Asset Pricing", "year": 2019, "page": 1, "section": "introduction", "sha256": "2f9e655c4aaab4f1", "role": "introduction", "genre": "mixed_theory_empirical", "subfield": "asset_pricing", "confidence": 0.8, "split": "train"}, {"unit": "intro_move", "score": 0.8637, "doc_id": "jof_12844", "doi": "10.1111/jofi.12844", "title": "Trading Against the Random Expiration of Private Information: A Natural Experiment", "year": 2020, "page": 3, "section": "introduction", "sha256": "649cb23d513bbe54", "role": "introduction", "genre": "mixed_theory_empirical", "subfield": "market_microstructure", "confidence": 0.8, "split": "train"}, {"unit": "intro_move", "score": 0.8193, "doc_id": "jof_12544", "doi": "10.1111/jofi.12544", "title": "Why Does Return Predictability Concentrate in Bad Times?", "year": 2017, "page": 3, "section": "introduction", "sha256": "9650ee6f9da03c5e", "role": "introduction", "genre": "mixed_theory_empirical", "subfield": "asset_pricing", "confidence": 0.8, "split": "train"}, {"unit": "identification", "score": 0.8183, "doc_id": "jof_12526", "doi": "10.1111/jofi.12526", "title": "Firm Investment and Stakeholder Choices: A Top-Down Theory of Capital Budgeting", "year": 2017, "page": 1, "section": NaN, "sha256": "a9f703cc3137b189", "role": "empirical_design", "genre": "theoretical", "subfield": "corporate_finance", "confidence": 0.8, "split": "train"}, {"unit": "intro_move", "score": 0.8148, "doc_id": "jof_12846", "doi": "10.1111/jofi.12846", "title": "Access to Collateral and the Democratization of Credit: France’s Reform of the Napoleonic Security Code", "year": 2020, "page": 2, "section": "introduction", "sha256": "b5dfbca07d3c9237", "role": "introduction", "genre": "empirical", "subfield": "corporate_finance", "confidence": 0.8, "split": "train"}, {"unit": "intro_move", "score": 0.8145, "doc_id": "jof_12544", "doi": "10.1111/jofi.12544", "title": "Why Does Return Predictability Concentrate in Bad Times?", "year": 2017, "page": 3, "section": "introduction", "sha256": "9650ee6f9da03c5e", "role": "introduction", "genre": "mixed_theory_empirical", "subfield": "asset_pricing", "confidence": 0.8, "split": "train"}], "profile": "empirical_all"}10{"ts": "2026-11-03T23:52:44+00:00", "task": "improve_abstract", "doc": "wp29", "domain": "out_of_domain", "condition": "E_full_engine", "rep": 1, "paper": "/Users/simon-pierreboucher/Desktop/UQO/Recherche/Recherche-UQO/methodes/autonomous-research-claude-code", "degraded_literature": "", "ok": true, "text": "Do the phases, gates and integrity rules in coding-agent research instructions change what a fixed agent delivers? We ran \\nMainRuns{} preregistered Claude Code runs (model \\texttt{claude-opus-5-5}) on \\nConfTasks{} bounded scientific tasks scored by hidden deterministic oracles. Each task received \\nRepeats{} fresh sessions under each of four regimes:\n\\begin{itemize}\n\\item an open instruction;\n\\item a structured protocol;\n\\item the protocol plus an executable gate;\n\\item these plus a review-and-repair session.\n\\end{itemize}\nIntervention-free verified completion was \\pA{} under the open instruction and \\pD{} under the full method (paired difference \\dDA{}; task-cluster bootstrap 95\\% interval [\\dDAlo, \\dDAhi]). The mechanism contrasts are equally null. Completion was already high under the open instruction, and the review loop was never triggered, so no correctness gain is detected. The gated arms reached full contract compliance at two to three times the inference cost. On inspection, every non-success was an oracle defect, and correcting them leaves the null unchanged. For bounded tasks with known answers, structure buys auditability, not correctness, and the scarce resource is a trustworthy oracle.", "change_log": "- **Move sequence:** question (with the gap folded in as \"agent held fixed\"), then design and contribution, then oracle protection, then the headline magnitude with its interval, then mechanism null, then cost and compliance, then the oracle diagnosis, then a bounded implication.\n- **Sentence count:** Eight sentences, slightly above the corpus median of 5. I kept the extra sentences because the contribution is a null result. A null is only credible if the abstract states the protection design and the oracle diagnosis.\n- **Question move:** I opened with a direct question rather than a motivation paragraph. This sharpens the question, as the brief asks.\n- **Contribution:** The contribution, a preregistered, agent-fixed controlled comparison, is carried by \"preregistered comparison … held fixed.\" It is based on the Introduction's contribution (iii).\n- **Unchanged facts:** All macros, the client and model versions, \"six families,\" \"four regimes,\" and \"two to three times\" are kept verbatim.\n- **Conventions not applied:** I did not apply finance-specific conventions such as identification language or economic-significance framing. They do not fit a methods/agent-evaluation paper.\n- **Length:** About 145–150 words, counting each macro as one word, to meet the brief's 100–150 limit.\n--- referee pass ---\n- **Audit status.** The draft introduced no new numbers and no citations, so nothing needed removing. The \"missing\" numbers flagged by the audit belong to other sections and are not expected in an abstract. All macros and stated magnitudes are kept exactly: \\pA, \\pD, \\dDA, the interval bounds, and \"two to three times.\"\n- **Length.** The abstract is now about 150 words, within the brief. To save space I dropped:\n - the opening descriptive sentence, folded into the question;\n - the client version, the session count and the \"separate host\" detail. These remain in the body.\n- **Identification framing (added).** The null is now stated with its two main limits, both taken from the Introduction and Conclusion:\n - completion was already high under the open instruction (ceiling);\n - the arm-D review loop never triggered, so D$-$C cannot test review.\n \n The wording changes from \"no effect\" to \"no correctness gain is detected,\" which matches the reported interval.\n- **Contract-compliance claim narrowed.** \"The protocol, gate and reviewer bought full compliance\" became \"the gated arms reached full contract compliance.\" The Introduction reports zero failures in arms C and D but \\nContractFailB{} failures in arm B. The original phrasing could be read as crediting the protocol alone.\n- **Oracle-defect claim.** The hedge \"on inspection\" is retained. The draft had dropped the corrected-rubric robustness statement; it is now restored as \"correcting them leaves the null unchanged.\"\n- **Typo.** \"95\\ interval\" in the existing abstract is corrected to \"95\\%.\"", "flags": "- **Prerequisite in the abstract.** The abstract does not mention that a launcher defect truncated continuation prompts (amendment A-006). The frozen comparison is therefore effectively one of system-prompt protocol texts. Decide whether this belongs in the abstract.\n- **\"Completion was already high under the open instruction.\"** This rests on the Conclusion's \"the ceiling under the open instruction was high.\" Please confirm the value of \\pA{} supports this phrasing.\n- **\"Gated arms.\"** Confirm that this means arms C and D only, and that \\nContractFailB{} $>0$, so that the narrowed compliance claim is accurate.\n- **Material dropped for length.** Confirm you accept omitting the fault, length-matched control, transfer and reflexive experiments. They are covered in the body, but the original abstract listed them.\n- **\"Preregistered.\"** This relies on the Introduction's \"preregistered and frozen with hashes.\" Confirm that a public registration exists, or else use \"frozen before the confirmatory runs.\"", "cost_usd": 0.3221, "latency_s": 90.2, "usage": {"input": 30189, "output": 10067}, "n_calls": 2, "model": "claude-opus-5-5", "effort": "medium", "metrics": {"n_words": 170, "preserved_rate": 0.014, "new_numbers": 0, "new_rate": 0.0, "n_author_year": 0, "n_unverifiable_citations": 0, "unverifiable_examples": [], "unknown_cite_keys": 0, "n_unsupported_claims": 0, "max_overlap_share": 0.0, "todo_placeholders": 0}, "retrieved": [{"unit": "intro_move", "score": 0.9633, "doc_id": "jof_12766", "doi": "10.1111/jofi.12766", "title": "Labor-Technology Substitution: Implications for Asset Pricing", "year": 2019, "page": 1, "section": "introduction", "sha256": "2f9e655c4aaab4f1", "role": "introduction", "genre": "mixed_theory_empirical", "subfield": "asset_pricing", "confidence": 0.8, "split": "train"}, {"unit": "intro_move", "score": 0.8637, "doc_id": "jof_12844", "doi": "10.1111/jofi.12844", "title": "Trading Against the Random Expiration of Private Information: A Natural Experiment", "year": 2020, "page": 3, "section": "introduction", "sha256": "649cb23d513bbe54", "role": "introduction", "genre": "mixed_theory_empirical", "subfield": "market_microstructure", "confidence": 0.8, "split": "train"}, {"unit": "intro_move", "score": 0.8193, "doc_id": "jof_12544", "doi": "10.1111/jofi.12544", "title": "Why Does Return Predictability Concentrate in Bad Times?", "year": 2017, "page": 3, "section": "introduction", "sha256": "9650ee6f9da03c5e", "role": "introduction", "genre": "mixed_theory_empirical", "subfield": "asset_pricing", "confidence": 0.8, "split": "train"}, {"unit": "identification", "score": 0.8183, "doc_id": "jof_12526", "doi": "10.1111/jofi.12526", "title": "Firm Investment and Stakeholder Choices: A Top-Down Theory of Capital Budgeting", "year": 2017, "page": 1, "section": NaN, "sha256": "a9f703cc3137b189", "role": "empirical_design", "genre": "theoretical", "subfield": "corporate_finance", "confidence": 0.8, "split": "train"}, {"unit": "intro_move", "score": 0.8148, "doc_id": "jof_12846", "doi": "10.1111/jofi.12846", "title": "Access to Collateral and the Democratization of Credit: France’s Reform of the Napoleonic Security Code", "year": 2020, "page": 2, "section": "introduction", "sha256": "b5dfbca07d3c9237", "role": "introduction", "genre": "empirical", "subfield": "corporate_finance", "confidence": 0.8, "split": "train"}, {"unit": "intro_move", "score": 0.8145, "doc_id": "jof_12544", "doi": "10.1111/jofi.12544", "title": "Why Does Return Predictability Concentrate in Bad Times?", "year": 2017, "page": 3, "section": "introduction", "sha256": "9650ee6f9da03c5e", "role": "introduction", "genre": "mixed_theory_empirical", "subfield": "asset_pricing", "confidence": 0.8, "split": "train"}], "profile": "empirical_all"}11