SPB Git

spb/wp45_uqo

Public
13.7 KB · 45 lines latex
1% ============================================================================2% Section 2 — Related literature (WP45 / JoF-250)3% All citation keys resolve to paper/references.bib; every entry was verified4% on 2026-11-03 (see paper/sections/literature_verification.md).5% ============================================================================6\section{Related literature}7\label{sec:literature}89This paper sits at the intersection of four literatures: the linguistic study of scientific genres, the empirical study of writing conventions and readability in economics and finance, retrieval-augmented and agentic uses of large language models (LLMs) in scholarly writing and review, and the evaluation of LLM-based research tools under contamination and reproducibility constraints. We review each in turn and then state what JoF-250 adds.1011\subsection{Genre analysis and rhetorical moves in scientific writing}12\label{sec:lit-genre}1314The notion that research articles follow a learnable rhetorical structure originates in genre analysis. \citet{swales1990genre} describes the research article as a genre whose introductions ``create a research space'' (CARS) through a small number of recurring moves: establishing a territory, establishing a niche, and occupying the niche; \citet{swales2004research} later revisits the model and extends it to other research genres. The CARS framework has since been operationalised as \emph{move analysis}, in which trained coders segment a text into functional units and record their frequency, order and linguistic realisation. \citet{samraj2002introductions} shows that move structure varies even between neighbouring subfields and that the original model requires elaboration to capture, among other things, how centrality claims and niche establishment are intertwined. \citet{kanoksilapatham2005rhetorical} extends the method beyond introductions by proposing a fifteen-move description of entire biochemistry articles, and \citet{lin2012structural}, in a cross-disciplinary sample of empirical articles, document that the canonical introduction--methods--results--discussion template is only one of several macro-structures actually in use. Two lessons follow for our purposes. First, rhetorical conventions are discipline- and even subfield-specific, which argues against importing a generic template into finance. Second, conventions are measurable: once an ontology of moves is fixed, their distribution can be estimated on a corpus rather than asserted.1516A parallel strand studies how authors modulate commitment and guide readers. \citet{hyland1996hedging,hyland1998hedging} analyses hedging in science research articles as a pragmatic resource for calibrating claims and anticipating objection, and \citet{hyland2004metadiscourse} reappraise metadiscourse---the text about the text through which writers organise arguments and engage readers. \citet{hyland2017informal} trace changes in informality markers across four disciplines over fifty years and find that the trends differ by field. These studies motivate the hedging, transition and claim--evidence dimensions of our annotation schema. Computational linguistics has long tried to automate such analyses. \citet{teufel2002summarizing} introduced argumentative zoning, assigning a rhetorical status (background, own contribution, contrast, and so on) to sentences of scientific articles, and \citet{cohan2019sequential} show that pretrained language models classify the rhetorical role of consecutive sentences in abstracts with high accuracy. Large open corpora of structured full text \citep{lo2020s2orc} and dedicated extraction tools such as GROBID \citep{lopez2009grobid} made document-level analysis practical, although finance articles---heavy in tables, footnotes and equations---remain a demanding parsing target.1718Within economics, writing guidance has mostly taken the form of essays by senior scholars rather than corpus studies. \citet{mccloskey1985economical} offers rules of economical writing in \emph{Economic Inquiry}, and \citet{thomson1999guide} gives a detailed guide to writing economic theory in the \emph{Journal of Economic Literature}. Such guidance is valuable but prescriptive: it reflects experience and taste, not measured practice in a specific journal. Our profiles are intended to complement it with descriptive estimates, while remaining explicit that observed frequency is not evidence of editorial superiority.1920\subsection{Writing conventions and readability in economics and finance}21\label{sec:lit-finance}2223Finance has an unusually developed empirical literature on \emph{writing}, albeit mostly about corporate disclosures rather than research articles. \citet{li2008readability} measures the readability of annual reports with the Fog index and document length and relates it to current earnings and earnings persistence. \citet{loughran2011liability} show that general-purpose sentiment dictionaries misclassify a large share of financial vocabulary and build finance-specific word lists; \citet{loughran2014readability} argue that the Fog index is poorly specified for 10-Ks and propose file size as a simpler readability proxy. \citet{loughran2016textual} survey textual analysis in accounting and finance, and \citet{gentzkow2019text} review text-as-data methods across economics. These papers are relevant to JoF-250 in two ways: they establish that domain-specific lexicons and measurement choices matter for financial text, and they furnish deterministic measures (lengths, lexical features, formula-based readability) that can be computed on research articles as well as on filings.2425A smaller literature treats the economics research article itself as an object of measurement. \citet{card2013nine} document, among nine trends in the top-five economics journals since 1970, that published papers have become roughly three times longer and have more coauthors, and \citet{card2014pagelimits} study how two journals' page limits changed submission length and composition. \citet{hengel2022publishing} applies readability scores to abstracts in top economics journals and finds that female-authored papers are better written and that the gap widens during peer review, evidence that writing standards are enforced in review. \citet{dowling2018easy} ask whether more readable finance articles attract more citations. Outside economics, \citet{plaven2017readability} find a steady decline in the readability of scientific abstracts across more than 700,000 abstracts since 1881, and \citet{hartley2014structured} summarises research on structured abstracts, which tend to be more informative than unstructured ones. Taken together, this work shows that section lengths, abstract organisation and readability are measurable properties of research articles that vary across journals and over time, but no study we are aware of estimates the distribution of \emph{rhetorical} conventions---contribution placement, identification framing, robustness presentation---for a finance journal. More recently, \citet{korinek2023generative} catalogues use cases of generative AI across the research workflow, including writing and editing, and anticipates that economists will increasingly rely on such tools; this motivates asking what conventions those tools should be calibrated to.2627\subsection{Retrieval-augmented generation and LLM-assisted writing and review}28\label{sec:lit-rag}2930Retrieval-augmented generation (RAG) combines a parametric language model with a non-parametric memory accessed through a retriever. \citet{lewis2020rag} introduce the canonical formulation for knowledge-intensive tasks, \citet{guu2020realm} integrate retrieval into pre-training, and \citet{karpukhin2020dpr} show that dense passage retrieval learned from question--passage pairs outperforms sparse baselines in open-domain question answering. \citet{gao2023ragsurvey} survey the subsequent proliferation of ``naive'', ``advanced'' and ``modular'' RAG designs. Virtually all of this work retrieves \emph{content}---passages whose facts are to be reproduced. The JoF-250 engine retrieves something different: structural annotations and summaries that describe \emph{how} an argument is organised, with excerpts limited to what is necessary and every result linked to provenance. Condition C in our experiment is a conventional content-RAG baseline precisely so that this distinction can be tested rather than assumed.3132LLMs are already used for scholarly writing and review at scale. \citet{liang2024mapping} estimate, with a population-level statistical framework applied to almost a million papers, a sharp rise in LLM-modified text after late 2022, and \citet{kobak2025delving} use excess-vocabulary analysis of PubMed abstracts to infer that a substantial fraction of 2024 biomedical abstracts were processed with LLMs. \citet{liang2024monitoring} apply the same framework to peer reviews at AI conferences, and \citet{latona2024lottery} report, in a quasi-experimental study of ICLR 2024, that AI-assisted reviews are associated with higher scores. \citet{else2023abstracts} and \citet{gao2023comparing} show that blinded reviewers identify only part of the ChatGPT-generated abstracts shown to them, and \citet{vannoorden2023survey} report mixed expectations among 1,600 surveyed researchers. On the usefulness side, \citet{liang2024feedback} find in a large-scale analysis that GPT-4 feedback on manuscripts overlaps with human reviewer comments and is judged helpful by many authors, whereas \citet{du2024critique} show that LLM-generated reviews of NLP papers are often deficient in ways experts readily spot. \citet{thelwall2024chatgpt} finds that single ChatGPT research-quality scores correlate only weakly with an expert's own ratings, with averaging over repeated runs improving agreement, and \citet{hosseini2023fighting} discuss the integrity risks of LLM-assisted peer review. \citet{si2024ideas} run a controlled comparison of LLM and expert research ideas, a reminder that evaluating LLM research assistance requires blinding and controls. \citet{shumailov2024collapse} show that training recursively on model-generated text degrades models, one argument for anchoring writing tools in human-written, pre-LLM text.3334Because our primary outcome is scored by an LLM judge, the literature on LLM-as-a-judge bears directly on our design. \citet{zheng2023judge} document position, verbosity and self-enhancement biases and show that strong judges can nevertheless agree with human preferences at human-level rates on their benchmarks. \citet{wang2024notfair} show that the ordering of candidates can flip pairwise judgments, and \citet{panickssery2024self} link self-preference to a model's ability to recognise its own output. We therefore randomise presentation, blind the judge to condition, and treat judge scores as a supplement to deterministic fidelity metrics and to human ratings where available.3536\subsection{Evaluation, contamination and reproducibility of LLM-based research tools}37\label{sec:lit-eval}3839Any experiment in which a foundation model is evaluated on published articles must confront the possibility that those articles were in its training data. \citet{magar2022contamination} distinguish memorisation from exploitation of contaminated data, \citet{sainz2023contamination} argue that contamination must be measured benchmark by benchmark, and \citet{golchin2024timetravel} and \citet{deng2024investigating} propose completion-based probes for detecting it. \citet{carlini2023memorization} quantify how memorisation grows with model size, duplication and prompt length, and \citet{lee2022dedup} show that deduplicating training data reduces verbatim regurgitation. These findings motivate our contamination statement, the withholding of the published introduction from research briefs, and the overlap audit between generated text and the corpus. More broadly, \citet{kapoor2023leakage} document leakage as a pervasive source of irreproducibility in machine-learning-based science and propose model information sheets; \citet{bowman2021benchmarking} discuss what credible benchmarks require; \citet{dodge2019showyourwork} call for reporting budgets and variance rather than best runs; and \citet{pineau2021reproducibility} report on the NeurIPS reproducibility programme and checklist. \citet{nosek2018preregistration} make the general case for preregistration: specifying hypotheses, outcomes and analyses before data are collected. Our protocol follows these recommendations by freezing document-level splits, preregistering hypotheses and primary outcomes, logging tokens, cost, latency and model versions, and reporting failed runs.4041\subsection{Positioning of this paper}42\label{sec:lit-position}4344JoF-250 differs from the work above in four respects. First, its writing conventions are \emph{corpus-derived and measured}: rather than prescriptive advice \citep{mccloskey1985economical,thomson1999guide}, we estimate medians, interquartile ranges and sample sizes for rhetorical moves and structural features on 276 pre-ChatGPT \emph{Journal of Finance} articles, using a move ontology in the tradition of \citet{swales1990genre} and \citet{kanoksilapatham2005rhetorical}. Second, profile construction is separated from evaluation by frozen, document-level held-out splits, and contamination is acknowledged and probed rather than ignored \citep{sainz2023contamination,kapoor2023leakage}. Third, the engine's retrieval targets rhetorical structure rather than content, and its outputs are subjected to deterministic fidelity audits---numerical preservation, citation verification, corpus overlap---so that improvements in judged quality can be checked against unchanged facts, addressing the known weaknesses of LLM judges \citep{zheng2023judge,wang2024notfair}. Fourth, the engine is implemented as a tested Claude Code extension with a preregistered benchmark, so that the paper studies a system that is independently usable for future manuscripts.45
Files · abe8945aefffa0096080b1470e6c1c6a089f5faa
Menu

Theme