SPB Git

spb/wp45_uqo

Public
28.0 KB · 130 lines latex
1% paper/sections/architecture.tex — WP45 / JoF-2502\providecommand{\TODOmacro}[1]{\textcolor{red}{[TODO macro: \texttt{#1}]}}3\providecommand{\profval}[1]{\ifcsname #1\endcsname\csname #1\endcsname\else\TODOmacro{#1}\fi}4\providecommand{\profvalc}[1]{\ifcsname #1\endcsname\csname #1\endcsname\else\textcolor{red}{\scriptsize TODO}\fi}  % compact fallback for table cells; macro name stays in the source56\section{JoF-250 architecture and Claude Code implementation}7\label{sec:architecture}89\subsection{Overview}10\label{sec:arch-overview}1112JoF-250 is a local-first pipeline implemented as a Python package (\texttt{jof250}, Python 3.12) with a command-line interface (Typer) and a thin Claude Code layer on top. Table~\ref{tab:pipeline} lists its stages, their inputs and outputs, and the artefacts each stage leaves behind; every artefact carries a timestamp, the git code version (with a \texttt{-dirty} suffix when the working tree had uncommitted changes), and the host description. The design choices follow from the constraints of the mandate: the corpus is copyrighted and must stay on one machine; the system must be usable on a single Mac without network services; every number in this paper must be traceable to a recorded artefact; and the engine must be useful for manuscripts other than this one. Only two components call an external service: the annotation and generation stages call the Claude API, and the citation audit calls the Crossref API. Everything else, including embeddings, runs locally.1314\begin{table}[htbp]15\centering16\small17\caption{Pipeline stages of JoF-250. Modules are in \texttt{src/\allowbreak{}jof250/\allowbreak{}}; artefacts marked \textit{private} are excluded from version control.}18\label{tab:pipeline}19\begin{tabularx}{\linewidth}{>{\raggedright\arraybackslash}p{0.17\linewidth}>{\raggedright\arraybackslash}p{0.21\linewidth}X}20\toprule21Stage (module) & Input & Output / artefacts \\22\midrule23Inventory (\texttt{inventory}) & PDF folder, \texttt{index.\allowbreak{}csv}, Crossref & \texttt{corpus\_\allowbreak{}manifest.\allowbreak{}parquet}, \texttt{corpus\_\allowbreak{}audit.\allowbreak{}md}, frozen manifest, inventory ledger \\24Extraction (\texttt{parsing}) & included PDFs & one structured JSON per document (\textit{private}), \texttt{parse\_\allowbreak{}summary.\allowbreak{}parquet}, parse ledger \\25Segmentation (\texttt{segmentation}) & parsed JSON & rule-based section roles; bounded annotation payload (rights cap) \\26Annotation (\texttt{rhetoric}, \texttt{llm}) & payload, prompt v1.0, Claude API & schema-validated annotation per document, \texttt{annotations\_\allowbreak{}table.\allowbreak{}parquet}, API ledger, response cache \\27Splits (\texttt{scripts/\allowbreak{}03}) & manifest & \texttt{heldout\_\allowbreak{}v1.\allowbreak{}json} (train / held-out document identifiers) \\28Profiles (\texttt{features}) & parsed JSON, annotations, train split & \texttt{profiles/\allowbreak{}*.\allowbreak{}yaml}, \texttt{features.\allowbreak{}parquet}, figures \\29Index (\texttt{embeddings}) & parsed JSON, annotations & \texttt{units.\allowbreak{}parquet}, \texttt{embeddings.\allowbreak{}npy}, \texttt{index\_\allowbreak{}meta.\allowbreak{}json}, 8-gram shingles \\30Retrieval (\texttt{retrieval}) & query, filters & ranked units with provenance \\31Engine (\texttt{generation}) & manuscript, profile, retrieved units, Claude API & proposal file, report (change log, flags, audit, diff), engine-run ledger \\32Audits (\texttt{auditing}) & manuscript, bibliography, Crossref & deterministic audit report; evidence lookup \\33Evaluation (\texttt{evaluation}) & fixtures, protocol, Claude API & \texttt{runs.\allowbreak{}jsonl}, \texttt{judgments.\allowbreak{}jsonl}, \texttt{summary.\allowbreak{}json} \\34Integration (\texttt{integrations.\allowbreak{}claude\_\allowbreak{}code}) & command, agent and skill definitions & \texttt{.\allowbreak{}claude/\allowbreak{}\{commands,agents,skills\}} (project scope) or \texttt{\textasciitilde/\allowbreak{}.\allowbreak{}claude/\allowbreak{}} (user scope, on request) \\35\bottomrule36\end{tabularx}37\end{table}3839\subsection{Measured writing profiles}40\label{sec:arch-profiles}4142A profile is a YAML file that records, for a named subset of the training split, the empirical distribution of structural and rhetorical features. The structural features are computed from the parsed documents (\texttt{features.\allowbreak{}py}): page count; words in the body and in the appendix; abstract words; numbers of sections, subsections, paragraphs, equations, tables, figures, footnotes, reference entries, and in-text citations; words and paragraphs of the introduction and its share of the body; words and share of the conclusion; the body shares of sections whose rule-based role is theory, data, results, or robustness; words per table; references and equations per thousand words; and indicator variables for the presence of a robustness section, a separate literature section, an in-text appendix, and an Internet Appendix. For each numeric feature the profile stores the sample size, median, quartiles, mean, minimum, maximum, and a 95\% bootstrap interval for the median (500 resamples, seed 45). When the subset contains annotated documents, a \texttt{rhetoric} block adds the distribution of genres, subfields, and methods; the share of abstracts in which each of the six abstract moves is present, and the median sentence position of each; the number of abstract sentences; the distribution of the first introduction move and of the most common move sequences (consecutive duplicates collapsed); the position of the contribution paragraph, absolute and relative to the length of the introduction; the share of introductions with a roadmap according to the model and according to the regular expression; the location of literature positioning; contribution types; the causal-language score; results-presentation conventions; robustness location and types; standard-error and unit conventions in table notes; uncertainty in figures; the conclusion flags; hedging; the share of claims with a stated evidence type; and, as diagnostics, the model--rule role agreement and the model's own confidence.4344Profiles are stratified by selection rules on genre and subfield (Appendix~\ref{app:profiles}): an all-document profile (\texttt{corpus\_\allowbreak{}all}), an all-empirical profile, and subfield profiles for empirical asset pricing, corporate finance, financial intermediation and banking, household and behavioural finance, financial econometrics, and theoretical finance. Because the subfield of a document is an annotation output, the subfield profiles depend on the annotation run; with the full run, the training split yields \profCorpusAllN{} documents in \texttt{corpus\_\allowbreak{}all}, \profEmpiricalAllN{} in \texttt{empirical\_\allowbreak{}all}, \profEmpiricalAssetPricingN{} in \texttt{empirical\_\allowbreak{}asset\_\allowbreak{}pricing}, \profCorporateFinanceN{} in \texttt{corporate\_\allowbreak{}finance}, \profFinancialIntermediationBankingN{} in \texttt{financial\_\allowbreak{}intermediation\_\allowbreak{}banking}, \profHouseholdBehavioralFinanceN{} in \texttt{household\_\allowbreak{}behavioral\_\allowbreak{}finance}, \profTheoreticalFinanceN{} in \texttt{theoretical\_\allowbreak{}finance}, and \profFinancialEconometricsN{} in \texttt{financial\_\allowbreak{}econometrics} (Appendix~\ref{app:profiles}). The engine falls back from a subfield profile with fewer than eight documents to the all-empirical profile and then to the all-document profile. To give one illustration of what a profile summary conveys, in the \texttt{empirical\_\allowbreak{}all} profile the annotator finds a results move in \profEmpiricalAllAbsResults\% of abstracts but an explicit magnitude in only \profEmpiricalAllAbsMagnitude\%, the first move of the introduction is \profEmpiricalAllFirstMove{} in \profEmpiricalAllFirstMoveShare\% of articles, the contribution paragraph sits at a median relative position of \profEmpiricalAllContribPos{} (interquartile range \profEmpiricalAllContribPosQa{}--\profEmpiricalAllContribPosQb{}) in the introduction, a roadmap paragraph is present in \profEmpiricalAllRoadmap\% of articles according to the model and \profEmpiricalAllRoadmapRegex\% according to the regular expression, the median causal-language score is \profEmpiricalAllCausal{} on the 1--5 scale, and economic significance is discussed in \profEmpiricalAllEconSig\% of articles. Figures~\ref{fig:abstract-moves} to~\ref{fig:moves-robustness} show the corresponding distributions for the training split. Every profile file carries the same caveat in its header, which the engine repeats to the model in every prompt that uses a profile: observed frequencies describe what published articles did in 2017--2020; they are not evidence that these choices are editorially superior or required. The point is not rhetorical modesty. A convention can be frequent because it is useful, because it is enforced, or because it is habitual, and a corpus of accepted articles cannot distinguish these cases; the profiles are offered as a measured description that an author can consult and deviate from, and the engine's prompts ask the model to say when it deviates.4546\begin{figure}[htbp]47\centering48\IfFileExists{figures/fig_abstract_moves.pdf}{\includegraphics[width=\linewidth]{figures/fig_abstract_moves.pdf}}{\fbox{\parbox{0.9\linewidth}{\centering\small Figure \texttt{fig\_\allowbreak{}abstract\_\allowbreak{}moves.\allowbreak{}pdf} is produced by \texttt{jof250 profiles build}.}}}49\caption{Share of abstracts in which each of the six abstract moves (question, gap, method, results, magnitude, contribution) is present, by annotated genre, training split only; genres with fewer than five annotated documents are omitted. Presence is judged by the model annotator (prompt \texttt{document\_\allowbreak{}annotation} v1.0) from the abstract text and has not been validated by human coders; the deterministic magnitude check of Section~\ref{sec:extraction-checks} is reported in the profile files alongside the model's judgement.}50\label{fig:abstract-moves}51\end{figure}5253\begin{figure}[htbp]54\centering55\IfFileExists{figures/fig_contribution_position.pdf}{\includegraphics[width=0.75\linewidth]{figures/fig_contribution_position.pdf}}{\fbox{\parbox{0.9\linewidth}{\centering\small Figure \texttt{fig\_\allowbreak{}contribution\_\allowbreak{}position.\allowbreak{}pdf} is produced by \texttt{jof250 profiles build}.}}}56\caption{Relative position of the contribution paragraph within the introduction (paragraph index divided by the number of introduction paragraphs), training split, validly annotated documents in which the annotator identified a contribution paragraph. The paragraph index is the model annotator's; paragraph boundaries come from the heuristic parser.}57\label{fig:contribution-position}58\end{figure}5960\begin{figure}[htbp]61\centering62\IfFileExists{figures/fig_moves_robustness.pdf}{\includegraphics[width=\linewidth]{figures/fig_moves_robustness.pdf}}{\fbox{\parbox{0.9\linewidth}{\centering\small Figure \texttt{fig\_\allowbreak{}moves\_\allowbreak{}robustness.\allowbreak{}pdf} is produced by \texttt{jof250 profiles build}.}}}63\caption{Left: the first rhetorical move of the introduction (six most frequent values). Right: where robustness checks are presented (six most frequent values). Training split, validly annotated documents; both classifications are the model annotator's under the controlled vocabularies of Table~\ref{tab:vocab}, and ``mixed'' denotes robustness material spread across several locations.}64\label{fig:moves-robustness}65\end{figure}6667\subsection{Private hybrid retrieval}68\label{sec:arch-retrieval}6970The retrieval index (\texttt{embeddings.\allowbreak{}py}, \texttt{retrieval.\allowbreak{}py}) is built from \emph{units} rather than from passages of full text. Seven unit types are indexed. Four are derived from the annotations and contain no quoted text: a document summary (title, genre, subfield, method, the annotator's structural summary, and keywords), an identification summary (estimand, strategy, assumptions, threats, causal-language score), one claim--evidence unit per annotated claim, and one introduction-move unit per introduction paragraph with at least one annotated move. Three contain short excerpts capped at 400 characters, which the rights policy permits: the opening of each section with its role, length, and subsection headings; the title and notes of each table; and figure captions (at most eight per document). Each unit records the document identifier, the SHA-256 digest of the source PDF, DOI, year, title, article type, page, section, role, genre, subfield, method, the annotator's confidence, and the split of the document. The current index holds \idxUnits{} units from \idxDocs{} documents (\idxDocsAnn{} with an annotation): \idxUnitSectionOpening{} section openings, \idxUnitTableNotes{} table-note units, \idxUnitFigureCaption{} captions, \idxUnitIntroMove{} introduction-move units, \idxUnitClaimEvidence{} claim--evidence units, \idxUnitDocumentSummary{} document summaries, and \idxUnitIdentification{} identification summaries.7172Queries are scored by a hybrid of BM25 \citep{robertson2009bm25} over a simple alphanumeric tokenisation and cosine similarity between normalised embeddings computed locally with a MiniLM sentence encoder \citep{reimers2019sbert, wang2020minilm} of dimension \idxDim{}; the two scores are min--max normalised within the filtered candidate set and combined with equal weight (the weight, and pure BM25 or pure dense modes, are parameters). Metadata filters restrict the candidates by split (training only by default), year, genre, subfield, section role or heading, and unit type; the \texttt{annotation} unit filter selects the four structural unit types and the \texttt{raw} filter the three excerpt types, which is how the conventional-RAG and structured-retrieval conditions of Section~\ref{sec:design} are distinguished. To avoid a single article dominating the context, at most two units per document are returned. Every hit carries its provenance record, which the engine writes into its reports so that a user can trace any retrieved pattern to a document, page, and section.7374\subsection{The writing engine}75\label{sec:arch-engine}7677The engine (\texttt{generation.\allowbreak{}py}) operates on a manuscript directory or file. The loader (\texttt{auditing.\allowbreak{}load\_\allowbreak{}manuscript}) accepts LaTeX projects, locating \texttt{paper/\allowbreak{}main.\allowbreak{}tex} or \texttt{main.\allowbreak{}tex}, stripping comments, resolving \texttt{\textbackslash input} and \texttt{\textbackslash include} recursively, reading the UQO template's \texttt{\textbackslash WP\ldots} metadata macros, splitting the text into sections and subsections, and extracting tables, figures, equations, labels, cross-references, and citation commands; it also scans the project's BibTeX files for keys and DOIs. Markdown manuscripts are accepted with the same section logic. Each section receives a rule-based role (introduction, literature, theory, data, empirical design, results, robustness, conclusion, appendix) so that a request for ``the results section'' can be resolved against arbitrary headings. A cheap lexical heuristic then detects the manuscript's genre (theoretical versus empirical) and subfield from the title, abstract, and headings, without an API call, and the profile is selected by the rules of Section~\ref{sec:arch-profiles}; the user may override the profile. For manuscripts outside the corpus's core genre (theory, methods, computer science, real estate), the engine injects a genre note instructing the model to apply the corpus conventions only where the argument structure is comparable and to say in the change log which conventions it did not apply.7879The material handed to the model is bounded to 60,000 characters: title, abstract, metadata, an optional verified research brief, the list of sections with roles and lengths, the current text of the target section (up to 12,000 characters, or a note that it is absent), the other sections truncated proportionally, the captions and number counts of up to twenty tables, up to twenty equations, and the list of available BibTeX keys. The task is one of three modes: \emph{outline} (a paragraph-by-paragraph plan with the rhetorical move and the evidence of each paragraph), \emph{write}, or \emph{upgrade} (rewrite an existing section preserving every number, citation, claim, and result). Conditions A to E of the experimental design are implemented as prompt families (Appendix~\ref{app:prompts}) that share one system prompt with hard constraints (never invent results, numbers, or citations; use only listed BibTeX keys or a visible \texttt{\textbackslash todo} placeholder; preserve verified facts; never copy sentences or imitate an author; return the section text followed by a change log and a list of flags). Condition A adds a one-line instruction; B a strong academic-writing instruction; C adds six raw excerpts retrieved by similarity from the training split, in the manner of conventional retrieval-augmented generation \citep{lewis2020rag}; D adds the profile summary (sample size, caveat, length medians, abstract and introduction move patterns, contribution placement, identification, results, robustness, table and conclusion conventions) together with six structured units with provenance, and a four-step procedure (identify what the evidence supports, choose the sequence of moves and state it in the change log, write, list unsupported statements under flags). Condition E, the default engine, runs D and then two further stages: a deterministic audit of the draft (numeric preservation against the material, citation keys against the bibliography, unsupported-claim sentences, and eight-word overlap with the corpus), and a specialist-referee pass in which the model receives the material, the draft, and the audit and must return a revised draft that fixes every audit item while keeping every verified fact, strengthening the methodological framing, and replacing unsupported statements with placeholders. The audit is re-run on the revised text. Ablations of E switch off retrieval, the profile, the referee pass, or the fact-check individually.8081The engine never edits the manuscript. Its output is a new file whose header records the mode, section, condition, model, timestamp, and manuscript hash and states that the file is a proposal, together with a sibling report containing the change log, the flags, the retrieval provenance, the deterministic audit, and a unified diff against the current section. Each invocation is appended to an engine-run ledger with its cost, latency, token usage, number of API calls, prompt versions, and detected genre.8283\subsection{Reviews, audits, and evidence lookup}84\label{sec:arch-audits}8586Three further commands reuse the loader. \texttt{jof250 review} produces role-based reviews from five prompts (referee, writing editor, table-and-figure editor, citation and integrity auditor, replication auditor; Appendix~\ref{app:prompts}); each receives the manuscript material, the deterministic audit facts, and, by default, the measured conventions of the selected profile with the genre note, and is instructed to tie every comment to a section, table, or equation and to leave every number unchanged. \texttt{jof250 audit} makes no API call: it checks that tables have captions and notes and are referenced, that labelled equations and figures are referenced and that every cross-reference resolves, that every citation key exists in the bibliography, and that the DOIs in the bibliography resolve through Crossref (cached, at most sixty per run); it lists sentences with strong claim verbs that carry no citation, cross-reference, number, or hedge; and it reports the maximum share of the manuscript's eight-word shingles found in any single corpus document, computed from locally stored hashes. The limits of these checks are stated in \texttt{docs/\allowbreak{}LIMITATIONS.\allowbreak{}md}: numeric preservation compares sets of numbers, not their meaning; unsupported-claim detection is a lexical heuristic; and citation verification establishes that a key and a DOI exist, not that the cited work supports the claim. \texttt{jof250 evidence} locates, for a claim given as text, the tables whose bodies contain the claim's numbers or keywords and the sentences that mention them, and returns a status of supported by a table, mentioned in text, or no evidence found.8788\subsection{Claude Code integration}89\label{sec:arch-claude-code}9091The host ran Claude Code version 2.1.317, which supports project-scoped slash commands as Markdown files with YAML front matter under \texttt{.\allowbreak{}claude/\allowbreak{}commands/\allowbreak{}}, sub-agents under \texttt{.\allowbreak{}claude/\allowbreak{}agents/\allowbreak{}}, and skills under \texttt{.\allowbreak{}claude/\allowbreak{}skills/\allowbreak{}}. The integration module generates these files from a single source of truth (\texttt{integrations/\allowbreak{}claude\_\allowbreak{}code.\allowbreak{}py}) so that the project copy and an optional user-scope copy cannot drift. Ten commands are installed (Table~\ref{tab:commands}): status, corpus audit, profile explanation, outline, write, review, upgrade, audit, evidence, and benchmark. The installed version passes the raw text typed after a command to the command file as \texttt{\$ARGUMENTS}; rather than relying on a parser that the version does not provide, each command file instructs the assistant how to parse its arguments (for example, \texttt{<paper-path> [--section <name>] [--condition X]}, with stated defaults), which CLI invocation to run, which output files to read, and what to show the user. The front matter restricts the tools each command may use to the project's CLI binary, a few read-only shell utilities, and file reading and searching, and the command bodies state that the manuscript is never modified and that proposals go to a \texttt{jof250\_\allowbreak{}proposals/\allowbreak{}} or \texttt{jof250\_\allowbreak{}reviews/\allowbreak{}} folder inside the user's paper directory. Seven specialist agents (Table~\ref{tab:agents}) wrap the review and audit commands with role-specific instructions and declared models; a skill file describes the engine, its rules for the assistant (always pass an explicit output path, relay every flag and audit item, state which conventions do not apply to non-finance manuscripts, warn that API calls cost money), and how to repair a missing environment. The project-scope installation is the default; because the author asked that the commands be available in every session, the CLI also offers a user-scope installation that copies only the files prefixed \texttt{jof-} or named \texttt{jof250} into the user's Claude Code directory and touches nothing else there, with documented removal. The commands were exercised from a separate sample research repository, as the mandate requires; the results of those exercises are reported with the engineering tests, not as experimental findings.9293\begin{table}[htbp]\centering\small94\caption{Slash commands installed under \texttt{.\allowbreak{}claude/\allowbreak{}commands/\allowbreak{}} and the CLI call each one wraps.}95\label{tab:commands}96\begin{tabularx}{\linewidth}{>{\raggedright\arraybackslash}p{0.2\linewidth}>{\raggedright\arraybackslash}p{0.33\linewidth}X}\toprule97Command & Arguments & Purpose \\ \midrule98\texttt{/\allowbreak{}jof-status} & \texttt{---} & Show JoF-250 pipeline status (corpus, parsing, annotations, profiles, index, spend) \\99\texttt{/\allowbreak{}jof-corpus-audit} & \texttt{---} & Re-run the JoF corpus inventory/verification and summarize the audit \\100\texttt{/\allowbreak{}jof-profile} & \texttt{[profile-name]} & Explain a measured JoF writing profile (empirical\_asset\_pricing, corporate\_finance, financial\_econometrics, theoretical\_finance, \ldots{}) \\101\texttt{/\allowbreak{}jof-outline} & \texttt{\textless{}paper-path\textgreater{} --section \textless{}name\textgreater{}} & Propose a section outline for a manuscript using JoF-250 (profiles + structured retrieval) \\102\texttt{/\allowbreak{}jof-write} & \texttt{\textless{}paper-path\textgreater{} --section \textless{}name\textgreater{} [--condition A|B|C|D|E]} & Draft a manuscript section with the full JoF-250 engine (profiles, structured retrieval, verification, specialist referee) \\103\texttt{/\allowbreak{}jof-review} & \texttt{\textless{}paper-path\textgreater{} [--role referee|editor|table\_\allowbreak{}editor|integrity|replication] [--full]} & Produce a referee-style review of a manuscript (roles: referee, editor, table\_editor, integrity, replication) \\104\texttt{/\allowbreak{}jof-upgrade} & \texttt{\textless{}paper-path\textgreater{} --section \textless{}name\textgreater{} [--preserve-results]} & Rewrite/upgrade an existing section while preserving every result, number and citation \\105\texttt{/\allowbreak{}jof-audit} & \texttt{\textless{}paper-path\textgreater{} [--tables] [--equations]} & Deterministic manuscript audit: tables, equations, citations/DOIs, unsupported claims, corpus overlap \\106\texttt{/\allowbreak{}jof-evidence} & \texttt{\textless{}paper-path\textgreater{} --claim ''...''} & Locate the evidence for a claim inside a manuscript (tables, numbers, sentences) \\107\texttt{/\allowbreak{}jof-benchmark} & \texttt{[--protocol heldout\_\allowbreak{}v1] [--dry-run]} & Run (or dry-run) the preregistered JoF-250 evaluation protocol \\108\bottomrule\end{tabularx}109\end{table}110111\begin{table}[htbp]\centering\small112\caption{Specialist agents installed under \texttt{.\allowbreak{}claude/\allowbreak{}agents/\allowbreak{}} (frontmatter \texttt{model} field as declared).}113\label{tab:agents}114\begin{tabularx}{\linewidth}{>{\raggedright\arraybackslash}p{0.27\linewidth}lX}\toprule115Agent & Model & Role \\ \midrule116\texttt{jof-corpus-auditor} & haiku & verifies the JoF-250 corpus inventory, manifest integrity, duplicates and bibliographic records \\117\texttt{jof-rhetorical-analyst} & sonnet & explains the rhetorical structure of a section/manuscript against the measured JoF profiles \\118\texttt{jof-writing-editor} & sonnet & proposes rewrites that keep every fact, number and citation unchanged \\119\texttt{jof-econometric-referee} & opus & identification, inference, robustness critique \\120\texttt{jof-table-figure-editor} & sonnet & captions, notes, units, uncertainty \\121\texttt{jof-integrity-auditor} & sonnet & citations, DOIs, unsupported claims, numeric consistency, overlap \\122\texttt{jof-replication-auditor} & sonnet & what a replicator needs and what is missing \\123\bottomrule\end{tabularx}124\end{table}125126\subsection{Provenance, ledger, and budget}127\label{sec:arch-provenance}128129Every API call goes through one wrapper (\texttt{llm.\allowbreak{}py}) that appends a record to \texttt{api\_\allowbreak{}ledger.\allowbreak{}jsonl} with the purpose, prompt identifier and version, model identifier as returned by the API, effort, input, output and cache token counts, cost computed from the list prices recorded in \texttt{config/\allowbreak{}models.\allowbreak{}yaml}, latency, stop reason, request identifier, cache key, and code version. Responses are cached by the hash of model, prompts, schema, effort, and output limit, so that re-running the annotation stage does not re-spend; generation and judging calls bypass the cache because nondeterminism is part of what the experiment measures. The wrapper refuses to place a call once cumulative spend reaches the budget cap fixed by amendment A-001 (\texttt{budget\_\allowbreak{}usd\_\allowbreak{}cap}), retries transient API errors with back-off, and records refusals. At the time of writing the ledger holds \apiCalls{} calls for \apiSpendUSD{}~USD, of which \apiSpendAnnotation{}~USD for annotation, \apiSpendAnnotationretry{}~USD for annotation retries, \apiSpendWrite{}~USD for engine writing calls, \apiSpendReferee{}~USD for referee passes, and \apiSpendReview{}~USD for review calls; benchmark spending is reported with the results. Engine invocations and benchmark runs are written to their own ledgers, and failed or refused runs are kept, not dropped. Together with the frozen manifest, the per-document hashes, the parser version stamped in every parsed file, and the seeds recorded in the split, profile, and fixture scripts, these ledgers allow every table and figure of this paper to be regenerated from recorded data by the commands listed in Appendix~\ref{app:reproducibility}.130
Files · 62a9da723bfa8125e943fa9f6b4bc31044d26d0e
Menu

Theme