SPB Git

spb/wp45_uqo

Public
9.9 KB · 18 lines latex
1\section{Introduction}\label{sec:introduction}23Scholarly writing in economics and finance follows conventions that are rarely written down. An introduction in a leading finance journal typically opens with a motivation, states a tension or a gap, previews the approach and the magnitude of the findings, places the contribution against a small set of neighbouring papers and closes with a roadmap; an abstract compresses the same arc into five sentences; results sections separate economic from statistical significance; conclusions bound their claims. Authors learn these conventions by reading, by being refereed and by imitation. Large language models (LLMs) learn them, if at all, by having read the same literature during pretraining, and are then prompted with generic advice (``write clearly, state the contribution'') whose relation to the practice of a given field is unknown.45This paper asks a narrow question about that gap. If one \emph{measures} the conventions of a field from a corpus written before conversational LLMs were widely available, and gives those measurements, together with provenance-bearing structural exemplars, to an LLM-based writing assistant, does the assistant write and review better than it does under a strong generic prompt or under conventional retrieval of similar text? And does it do so without copying the corpus or inventing facts? The question matters for two reasons. First, LLM assistance in academic writing has become common and is increasingly visible in published text \citep{liang2024mapping,kobak2025delving}; what such assistants are steered by deserves to be an object of study rather than an unexamined default. Second, retrieval-augmented generation is routinely proposed as the remedy for generic or hallucinated output \citep{lewis2020rag,gao2023ragsurvey}, yet in scholarly writing the thing to retrieve is unclear: raw sentences from other papers invite overlap and style imitation, whereas what an author needs is the \emph{structure} of a good argument and the discipline's conventions for presenting evidence.67We make three contributions. The first is a corpus-derived measurement of writing conventions. We assemble \nIncluded{} research documents from \emph{The Journal of Finance}, volumes 72--75 (2017--2020), all published before the public release of ChatGPT, verify every record against Crossref, and parse each PDF into sections, paragraphs, numbered equations, tables, figures, footnotes and references with a layout-aware extractor whose accuracy we report against an independent extraction (Section~\ref{sec:extraction}). We then annotate every document with a twelve-part rhetorical ontology---abstract moves, introduction moves and their order, contribution placement and type, identification framing and the strength of causal language, results presentation, robustness location, table and figure conventions, conclusion bounding, hedging and claim--evidence links---using a schema-validated LLM annotator with deterministic cross-checks (Section~\ref{sec:extraction}). From the \nTrain{}-document train split we compute measured \emph{writing profiles}: medians, interquartile ranges and bootstrap intervals for the structure and rhetoric of empirical, theoretical and subfield-specific articles (Section~\ref{sec:architecture}). Among the measurements: \profEmpiricalAllAbsResults\% of empirical abstracts state results and \profEmpiricalAllAbsMagnitude\% a magnitude; \profEmpiricalAllFirstMoveShare\% of empirical introductions open with a motivation move; the contribution paragraph sits at a median relative position of \profEmpiricalAllContribPos{} in the introduction; \profEmpiricalAllRoadmap\% of introductions end with a roadmap; the median causal-language strength is \profEmpiricalAllCausal{} on a five-point scale, against \profTheoreticalFinanceCausal{} for theory papers. We stress throughout that these are descriptions of practice, not editorial rules.89The second contribution is a usable system. JoF-250 is a local-first writing, review and audit engine: a hybrid retrieval index over \idxUnits{} structural units (document summaries, identification summaries, move-tagged introduction paragraphs, claim--evidence links and short excerpts, each with document, page and section provenance); a manuscript loader for LaTeX and Markdown projects; a generation pipeline that detects genre and subfield, selects a profile, retrieves rhetorical patterns, drafts, runs deterministic audits (numeric preservation, citation keys and DOI resolution, unsupported-claim detection, eight-gram overlap with the corpus) and passes the draft through a specialist-referee revision; and five reviewer roles. The engine is integrated into Claude Code, a terminal coding agent, as ten slash commands, seven specialist agents and one skill, so that it can be invoked on any manuscript, including this one and the author's other working papers. It never edits a manuscript: every output is a proposal with a change log, flags for the author, provenance and a diff (Section~\ref{sec:architecture}).1011The third contribution is a preregistered evaluation of the system against its alternatives (Section~\ref{sec:design}). Before any run, we froze five conditions---A, a bare prompt; B, a strong academic-writing prompt; C, B plus conventional retrieval of similar excerpts; D, B plus measured profiles and structured rhetorical retrieval; E, D plus deterministic verification and a specialist-referee pass---four ablations of E, eight writing and reviewing tasks, two held-out finance articles never used for profile construction, one out-of-domain manuscript, a blind LLM judge with a calibrated rubric, and an analysis plan with cluster-bootstrap intervals and Holm correction. The results (Section~\ref{sec:results}) are mixed in an instructive way. Profiles and structured retrieval alone do not move the judge's ratings of clarity, argument logic or contribution positioning relative to the strong prompt (H1) or to conventional retrieval (H2); the paired differences are small and their intervals include zero. The full engine does move them: relative to D, methodological rigor increases by \bmHEvsDMethodologicalRigorDiff{} points (95\% interval [\bmHEvsDMethodologicalRigorLo, \bmHEvsDMethodologicalRigorHi]) and fidelity by \bmHEvsDFidelityDiff{} points ([\bmHEvsDFidelityLo, \bmHEvsDFidelityHi]) on a seven-point scale, both surviving Holm correction (H3). The ablations (Section~\ref{sec:ablations}) attribute almost all of this to the specialist-referee pass: removing it costs \bmAbENoRefereeMethodologicalRigorDiff{} points of rigor, whereas removing retrieval, profiles or the deterministic fact-check changes little. The gain has a price: the referee pass makes the text more hedged, the judge rates E's output as slightly less useful than D's (\bmHEvsDUsefulnessDiff{} points, interval including zero), and E uses two generation calls, roughly twice the cost and latency. Sentences flagged as unsupported claims are rarer under E than under conventional retrieval, no condition produced a citation that could not be matched to the reference list, and generated text shares at most \bmOverlapOtherMax{} of its eight-grams with any corpus document other than the task's own source (H4 in part, H5). On the out-of-domain manuscript the finance profiles do not help, as expected.1213We read these findings as evidence about where the leverage lies in LLM-assisted scholarly writing. Telling a capable model what a field's introductions look like, or showing it structurally annotated exemplars, adds little that a strong prompt does not already elicit; the model appears to know the genre. Making the model \emph{check} its draft against the manuscript's evidence and against a referee's standards adds a great deal in the dimensions that matter for research integrity---rigor and fidelity---at roughly twice the cost and latency and with a small loss in the judge's rating of immediate usefulness, because a checked draft says less. The profiles retain a different value: they are inspectable, provenance-bearing descriptions of practice that a human author or editor can consult, and they define what the referee pass is asked to enforce.1415The study has clear limits, stated in Section~\ref{sec:discussion}. It is a pilot: two held-out finance documents, eight tasks, one to two repetitions, one out-of-domain manuscript; and its first run had to be discarded and repeated after two implementation defects were found during the write-up (Amendment A-004), which we document as part of the failure analysis. All quality ratings come from an LLM judge, whose self-consistency we measure but whose biases we cannot remove; the human-rating protocol is prepared but was not run. The generator and the judge may have seen the held-out articles in pretraining. The profiles describe one journal over four volumes. We therefore make no claim that the engine produces better papers than people do, nor that the \emph{Journal of Finance}'s practice is a standard to be imposed; we claim only what the preregistered contrasts support, and we release the pipeline so that the measurement can be repeated on other corpora and the evaluation extended with human raters.1617The paper proceeds as follows. Section~\ref{sec:literature} reviews genre analysis, writing conventions in economics and finance, retrieval-augmented generation, LLM-assisted writing and review, and the evaluation of LLM research tools. Section~\ref{sec:corpus} describes the corpus, inclusion rules and rights. Section~\ref{sec:extraction} presents the extraction pipeline and the annotation ontology. Section~\ref{sec:architecture} describes JoF-250 and its Claude Code integration. Section~\ref{sec:design} states the preregistered design. Sections~\ref{sec:results}--\ref{sec:human} report results, ablations and a failure analysis. Section~\ref{sec:discussion} discusses generalization, integrity and limitations; Section~\ref{sec:conclusion} concludes. Appendices give the ontology, prompts, parser audit, profiles, protocols and user manual.18
Files · f48490eca784017968beb6af9e04c942bd1390d9
Menu

Theme