SPB Git

spb/wp29_uqo

Public
10.8 KB · 62 lines latex
1\section{Corpus-informed method design}2\label{sec:corpus}34The protocol compared in arms B--D was derived from a content coding of the instruction files---the \texttt{CLAUDE.md} mandates---that the author had previously issued to Claude Code for earlier working papers in this series. The aim was to learn what a researcher writing such mandates repeatedly asks for and omits, and to turn the omissions into design elements. The exercise motivates the design; it makes no prevalence claim.56\subsection{Corpus and coding}78Thirteen files were retrieved from the author's git server on 2026-10-29 and hashed (SHA-256) into a manifest against which every file was re-verified before coding. They contain eleven distinct mandates (WP15, WP18--WP26 and WP28), one supplementary design brief (WP27), and one byte-identical duplicate of the WP15 mandate, which is recorded but counted once. The WP27 brief is a post-implementation description of a simulator rather than an instruction file and is excluded from every ``of 11'' count; WP27's own mandate could not be retrieved (HTTP 404) and nothing is inferred about it. The mandates range from 1,092 to 8,809 words (median 1,966); two (WP15 and WP28) are an order of magnitude longer than the rest, the extra length tracking engineering prescription rather than substantive research rules.910Coding followed a written codebook. Automated variables (length, headings, phase and gate labels, keyword families) were computed by a script from the file bytes; analyst variables were assigned in a single automated-plus-LLM-analyst pass after reading each document in full, each judgement recorded as \emph{yes}, \emph{partial} or \emph{no} with a verbatim quotation in the same cell so that it can be re-checked against the source. \emph{Partial} means the feature is addressed only indirectly, conditionally or without an operational mechanism. There was no human second coder, so no intercoder reliability is reported and none should be inferred. Table~\ref{tab:corpus} gives the counts.1112\begin{table}[htbp]13\centering14\small15\caption{Feature counts over the eleven distinct mandates (WP27 brief and the WP15 duplicate excluded). ``Partial'' is defined in the codebook as present but indirect, conditional or without an operational mechanism; ``---'' means the category was not used for that feature.}16\label{tab:corpus}17\begin{tabular}{>{\raggedright\arraybackslash}p{2.3in}ccc>{\raggedright\arraybackslash}p{2.05in}}18\toprule19Feature & Yes & Partial & No & Note \\20\midrule21Explicit research question (verbatim) & 10 & --- & 1 & WP22 states objectives, not a question \\22Explicit estimand or hypotheses & 7 & 4 & 0 & H1--H6 lists, theorem queues, primary outcome \\23Assumptions stated in the mandate & 2 & 8 & 1 & most require assumptions without stating them \\24Falsification / null-result conditions & 6 & 4 & 1 & \\25Literature / novelty requirement & 11 & 0 & 0 & universal; usually a ``novelty gate'' \\26Data-access controls & 7 & 4 & 0 & credentials, scraping, privacy, licensing \\27Temporal-information controls & 6 & 1 & 4 & all six data-based mandates; n/a for theory \\28Baseline / benchmark requirement & 8 & 3 & 0 & \\29Phases declared & 10 & --- & 1 & 5 to 13 phases; median 7 \\30Dependency / gate structure & 10 & 1 & 0 & ``partial'' = implicit ordering (WP21) \\31Stopping rules & 2 & 6 & 3 & ``partial'' = document and deliver partial result \\32Recovery / resume instructions & 2 & 6 & 3 & ``partial'' = progress logs, caching, retry limits \\33Provenance requirements & 9 & 2 & 0 & manifests with hashes in WP15, WP28 \\34Review / self-audit stage & 10 & 1 & 0 & results audits, simulated referees, red team \\35AI-use / disclosure statement & 1 & 0 & 10 & WP28 only \\36Budget / resource limits & 1 & 3 & 7 & WP26 asks for API-cost tiers; no numeric cap \\37Integrity prohibitions (``never invent \ldots'') & 11 & 0 & 0 & universal, usually in the opening paragraph \\38Pre-registration mentioned (keyword) & 4 & --- & 7 & no registry or timestamp mechanism \\39Random seeds mentioned (keyword) & 11 & --- & 0 & \\40Statements marked as added or reconstructed & 6 & --- & 5 & \\41\bottomrule42\end{tabular}43\end{table}4445\subsection{What the mandates consistently require}4647Eight patterns recur. (1) Every mandate opens by casting Claude Code as a multidisciplinary research team and immediately prohibits fabrication of data, results, citations, proofs or significance (11 of 11), the most stable element of the corpus. (2) Ten declare ordered phases (five to thirteen; median seven), nine make drafting conditional on validated results, and ten name gates, mostly as numbered acceptance gates. (3) All eleven require a verified literature review or novelty gate before any contribution claim. (4) Ten include audit or self-review as a phase: claim-to-file audits, an adversarial referee report, three simulated referees, or a red-team agent. (5) Nine require traceability of every number to a result file, registry or hashed manifest. (6) All six data-based mandates impose information-set discipline (look-ahead, as-of-date, vintage and chronological-split rules). (7) Eight name baselines. (8) Most provide an honest-failure fallback when work is blocked.4849Four excerpts show the register. The integrity preamble: ``Never fabricate data, results, citations, institutional details, variable definitions, statistical significance, sample sizes, or conclusions. Never write a result in the paper before it has been generated and verified from a saved machine-readable output.'' (\texttt{WP15\_CLAUDE.md}). A phase gate before drafting: ``do not start manuscript drafting until the baseline model is validated, claimed GE equilibria clear markets, principal simulations and exhibits reproduce, bibliography is verified, and theoretical claims have proofs or appropriate qualifications. If blocked, document the issue honestly and deliver a scoped partial result.'' (\texttt{WP18\_CLAUDE.md}). A pre-specified primary outcome: ``Pre-register primary endpoints and hypotheses before running final test evaluations. Primary outcome: out-of-time absolute log-price error on arm's-length residential transactions.'' (\texttt{WP26\_CLAUDE.md}). A declared acceptable null: ``A dead end with a mathematical explanation is better than a fabricated breakthrough. A strong negative result or counterexample may itself become the contribution.'' (\texttt{WP28\_CLAUDE.md}).5051\subsection{What the mandates consistently lack}5253The same documents are silent on mechanism. (1) Gates are declarative. Only WP15 and WP28 attach a gate to something that can fail mechanically---``The release script must fail if: a table references a missing result; a figure is missing source data; LaTeX does not compile; citations are unresolved; \ldots'' (\texttt{WP15\_CLAUDE.md})---and no mandate specifies who or what checks a gate other than the executing agent. (2) Verification is not independent. ``Independent audit'', ``independently checked'', ``independently verified'', ``hostile-referee critique'' and ``three independent review perspectives'' are all to be performed by the same agent; WP28 alone names the tension: ``Record unresolved issues candidly; do not have the same model merely declare its own work accepted.'' (\texttt{WP28\_CLAUDE.md}). (3) Pre-registration is not operational: four mandates ask for it, none specifies a registry, a timestamp, a hash or a freeze. (4) Stopping rules are rare: only WP15, with a dedicated stop-conditions section, and WP28 define when to stop; most others say to continue with feasible work. (5) Recovery is rare: only WP15 (resumable, idempotent, checkpointed jobs) and WP25 require it. (6) Budgets are absent: seven of eleven set no resource limit, only WP26 asks for an API-cost budget, and no mandate states a numeric token, dollar or compute cap. (7) Ten of eleven say nothing about disclosing AI involvement in the paper. (8) Success criteria are not quantified: targets read ``substantially improve'', ``publication-quality'' or ``the standard expected in a top finance journal'', none checkable from the document. (9) The instruction text itself has uncertain provenance: WP23 states that part of its protocol was reconstructed by an assistant after truncation, WP24 and WP26 carry sections marked as added on 2026-10-27, and WP18, WP20 and WP25 carry prefatory repository-mapping notes, so the mandate as retrieved is not always the mandate as issued.5455\subsection{What the corpus cannot establish}5657These documents are instructions: they record what the author asked Claude Code to do, not what was done. The coding therefore provides no evidence about how the earlier working papers were actually executed, whether the instructions were followed, or whether the papers were produced without human assistance; several mandates in fact anticipate human steps such as expert raters and provision of credentials. A mandate that requires a provenance manifest is coded as requiring one; whether the manifest exists is outside the coding. The corpus is also small and researcher-selected---eleven files by one author for one series---and was coded in a single pass without reliability statistics. Its counts motivate design patterns; they do not estimate the prevalence of any practice among autonomous-research mandates in general, and no inferential statistic is computed from them. Questions about execution require run-level evidence, which the experiment that follows is designed to produce.5859\subsection{From absences to design elements}6061The corpus shows a stable rhetorical protocol (role, integrity rules, phases, gates, audits, traceability) and an equally stable absence of mechanisms. Rather than add prose rules, arms B--D turn each absence into something checkable. Uncertain provenance of the instruction (absence 9) and non-operational pre-registration (absence 3) become a \emph{hashed, frozen mandate}: the instruction injected into each run is fixed and hashed before execution, so that what the executor received is known exactly. Declarative gates (absence 1) become \emph{executable gates} enforced by the controller in arms C and D, which can fail mechanically rather than by the agent's own judgement. Self-performed audits (absence 2) become \emph{external evaluators}: hidden deterministic oracles on a separate host, and in arm D a review-and-repair session separate from the executing session, so that no model merely declares its own work accepted. Missing budgets and stopping rules (absences 4 and 6) become \emph{explicit budgets and stop rules} in the protocol. The unrecorded human steps that several mandates anticipate become an \emph{intervention ledger} in which any human intervention during a run is recorded. And the audit phase that the mandates ask the same agent to perform becomes the \emph{separate review} that distinguishes arm D from arm C. Arm A, the open instruction, receives the task without any of these mechanisms, and so serves as the control against which their contribution is measured.62
Files · 4f59ed568641ceda6f39552d5cc3a441aa812088
Menu

Theme