% paper/sections/extraction_ontology.tex — WP45 / JoF-250 \providecommand{\TODOmacro}[1]{\textcolor{red}{[TODO macro: \texttt{#1}]}} \providecommand{\profval}[1]{\ifcsname #1\endcsname\csname #1\endcsname\else\TODOmacro{#1}\fi} \providecommand{\profvalc}[1]{\ifcsname #1\endcsname\csname #1\endcsname\else\textcolor{red}{\scriptsize TODO}\fi} % compact fallback for table cells; macro name stays in the source \section{Extraction pipeline and annotation ontology} \label{sec:extraction} \subsection{Parser choice} \label{sec:extraction-choice} The mandate asked that installed scientific PDF parsers be evaluated before one was chosen. On the host used for this study, GROBID \citep{lopez2009grobid} was not installed and the Docker daemon required to run it was not available, so a server-based parser was ruled out; the core of the system must in any case run on a single machine without network services. PyMuPDF (version 1.27) and poppler's \texttt{pdftotext} were available. We built the structural parser on PyMuPDF, which exposes the font name, font size, bounding box, and writing direction of every text span, and retained \texttt{pdftotext} as an independent second extraction for quality control (Section~\ref{sec:extraction-qa}). The parser is tuned to the journal's LaTeX template; it is not a general-purpose scientific parser, and its heuristics would need to be revisited for another journal. \subsection{Layout heuristics} \label{sec:extraction-heuristics} The parser (\texttt{parsing.\allowbreak{}py}, version \texttt{pymupdf-jof-heuristics-1.3}) proceeds in two passes over the text lines of a document. Every line is first normalised (Unicode NFKC, removal of non-printable characters, unification of the minus sign) and tagged with its dominant span's font, size, bold and italic attributes, and whether the font belongs to a mathematical family. Lines in the running head, the publisher's download watermark (a very small Times font in the margin), bare page numbers near the foot of the page, and narrow vertical fragments are discarded. The \emph{body font size} is estimated as the modal size of lines set in the template's roman text font; all subsequent thresholds are expressed relative to it. The front-matter pass identifies the title (bold lines at least 2.5 points larger than the body, on the first three pages), the author block (smaller non-bold lines between the title and the ``ABSTRACT'' marker), and the abstract (the small-type lines that follow the marker until body-size text resumes). The structural pass then walks the remaining lines and applies, in order, the following rules. A bold body-size line reading ``REFERENCES'' opens the reference list; a bold line beginning with ``Supporting Information'' records the page of the Internet Appendix pointer. Section headings are bold body-size lines that either begin with a roman numeral followed by a period (``\texttt{III.\allowbreak{}\ Data}'') or are centred on the page; headings beginning with ``Appendix'' switch the parser to appendix mode so that appendix words are counted separately. Subsection headings are italic body-size lines matching the template's ``\texttt{A.\allowbreak{}\ Title}'' pattern. Table captions are bold small-type lines matching ``\texttt{Table~\textit{roman}}'', followed by a bold title and small-type notes; landscape tables, whose lines are rotated by ninety degrees, are collected in a separate pass over rotated lines so that their captions and notes are not lost, and the ``Continued'' marker suppresses duplicate captions. Figure captions are bold-initial small-type lines matching ``\texttt{Figure~\textit{n}.}''; small-type lines located above a figure caption on the same page are treated as text inside the figure and ignored. Numbered equations are detected from their tag, a parenthesised number set flush with the right margin on a line in a mathematical or italic font or containing an operator; the equation text is reconstructed from the lines in the same vertical band. Footnotes are small-type lines in the lower part of the page, grouped into one footnote per leading number and per page. Reference entries are split using the hanging indent of the template: a line starting at the page's left margin begins a new entry, and each entry is parsed into an author string, a year, the remainder, and a DOI when present. Finally, body lines are grouped into paragraphs using the first-line indent, hyphenation across lines is repaired, and in-text author--year citations are collected with a regular expression. The output of the parser for one document is a JSON record holding the metadata, the abstract, the sections with their paragraphs and page numbers, subsections, equations, tables, figures, footnotes, references, in-text citations, page markers (references, appendix, supporting information), and summary counts; it is kept private because it contains the full text. \subsection{Confidence scores} \label{sec:extraction-confidence} Each parsed document carries a confidence score per component and an overall score. Title and authors receive 1 when found and 0 otherwise; the abstract receives 1 when its length lies between 400 and 2,500 characters, 0.5 when a shorter or longer abstract was captured, and 0 when none was; sections receive 1 when at least three roman-numbered headings were found in increasing order, 0.6 when two were found, and 0.2 otherwise; references receive 1 for at least ten entries and 0.5 for a non-empty list; equations and tables receive 0.9 when at least one was found and 0.7 otherwise. The overall score is the mean of the component scores. These scores are coarse by design: they are meant to route documents, not to measure accuracy. Over the included corpus the mean overall confidence is \parseMeanConf{}, and \parseLowConf{} documents fall below 0.8; these are the corrigenda and one-page notes, whose layout does not follow the article template and which are excluded by genre from the rhetoric profiles. \subsection{Parser quality assurance} \label{sec:extraction-qa} The mandate called for a manually checked gold-standard sample. No human annotator other than the author was available in the time frame of this study, and we did not want to present the author's own inspection as an independent gold standard. We therefore report an \emph{automated cross-parser agreement} check (\texttt{scripts/\allowbreak{}02\_\allowbreak{}parser\_\allowbreak{}qa.\allowbreak{}py}, results in \texttt{data/\allowbreak{}derived/\allowbreak{}parser\_\allowbreak{}qa.\allowbreak{}md}, reproduced per document in Appendix~\ref{app:parser-qa}): the structural counts produced by the PyMuPDF parser are compared with counts obtained by regular expressions on the layout-preserving output of \texttt{pdftotext}, on a stratified sample of \qaNSample{} documents (five randomly chosen articles per year, seed 45, plus every document of a non-article type). This is a comparison between two automated extractions, each with its own failure modes; agreement between them is evidence that the counts are not artefacts of one tool, but it is not an accuracy figure against human judgement, and we do not describe it as such. On this sample, the number of roman-numbered sections agrees exactly for \qaSectionsExact\% of documents, the number of tables agrees exactly for \qaTablesExact\% and within one for \profval{qaTablesWithinOne}\%, the number of figures agrees exactly for \qaFiguresExact\%, and the number of reference entries agrees within 10\% for \profval{qaReferencesWithinTenPct}\%. The abstract is found by both tools in \qaAbstractAgree\% of documents. Numbered-equation counts are the weak point: exact agreement is reached for only \qaEquationsExact\% of documents and agreement within 10\% for \profval{qaEquationsWithinTenPct}\%, because the two tools disagree on what constitutes an equation tag (the regular expression on the layout text also matches enumerated items and footnote markers in parentheses at the end of a line, while the PyMuPDF parser requires a right-margin position and a mathematical font). Equation counts should therefore be read as approximate throughout the paper. As a corpus-wide check of the front matter, the title extracted by the parser matches the Crossref title with a fuzzy similarity of at least 90 for \profval{qaTitleVsCrossrefGeNinetyAllDocs}\% of the \qaTitleNAllDocs{} included documents. The per-document table in Appendix~\ref{app:parser-qa} shows that the worst disagreements are concentrated in corrigenda, whose reference lists and headings do not follow the article template. \subsection{Annotation ontology} \label{sec:extraction-ontology} The mandate specified twelve families of properties to annotate, from the structure of the abstract to the boundedness of the conclusion, plus rhetorical moves, claim--evidence relations, hedging, and contribution types. We implemented them as a single document-level JSON schema (ontology version 1.0, \texttt{rhetoric.\allowbreak{}py}; full listing in Appendix~\ref{app:ontology}) whose enumerated fields use the controlled vocabularies of Table~\ref{tab:vocab}. Table~\ref{tab:ontology-map} maps each family of the mandate to the fields of the schema. Two families are deliberately annotated at a coarser level than the mandate's wording suggests. The \emph{literature} family is captured by the location of the literature positioning (inside the introduction, in a separate section, both, or none) and by the \texttt{related\_\allowbreak{}literature\_\allowbreak{}positioning} move, rather than by a graph of contrasts and lineage, because the bounded payload (Section~\ref{sec:corpus-rights}) does not contain the full literature discussion. The \emph{data} and \emph{empirical design} families are captured through section roles, the identification block (estimand, strategy, key assumptions, threats addressed, a 1--5 causal-language score, and a flag for explicit causal claims), and the table conventions block (units stated in notes, description of standard errors), rather than through variable lists. Theory papers are explicitly allowed not to follow the empirical template: the annotator is instructed to assign section roles honestly and to use the ``none'', ``unclear'', or ``not applicable'' values for empirical-only fields. The rhetorical-move inventory for introductions (motivation, tension or puzzle, literature gap, preview of approach, preview of results, contribution statement, mechanism or theory sketch, identification defence, robustness mention, roadmap, related-literature positioning, policy or broader implication, other) is an adaptation of move analysis in the tradition of genre studies \citep{swales1990genre, swales2004research} to the conventions of empirical finance, where the preview of results with magnitudes and the defence of identification are moves in their own right. Hedging is recorded as a three-level intensity with free-text devices, following the applied-linguistics literature on hedging in academic prose \citep{hyland1996hedging, hyland1998hedging}, rather than as a word count, since the payload is a sample of the text. \begin{table}[htbp]\centering\small \caption{Controlled vocabularies of the annotation ontology (version 1.0), as defined in \texttt{src/\allowbreak{}jof250/\allowbreak{}rhetoric.\allowbreak{}py}. Free-text and numeric fields are listed in Appendix~\ref{app:ontology}.} \label{tab:vocab} \begin{tabularx}{\linewidth}{>{\raggedright\arraybackslash}p{0.34\linewidth}>{\raggedright\arraybackslash}X}\toprule Field & Allowed values \\ \midrule \texttt{genre} & \texttt{empirical}, \texttt{theoretical}, \texttt{mixed\_\allowbreak{}theory\_\allowbreak{}empirical}, \texttt{methodological\_\allowbreak{}econometric}, \texttt{survey\_\allowbreak{}or\_\allowbreak{}address}, \texttt{replication\_\allowbreak{}or\_\allowbreak{}correction} \\ \texttt{subfield} & \texttt{asset\_\allowbreak{}pricing}, \texttt{corporate\_\allowbreak{}finance}, \texttt{financial\_\allowbreak{}intermediation\_\allowbreak{}banking}, \texttt{household\_\allowbreak{}finance}, \texttt{market\_\allowbreak{}microstructure}, \texttt{international\_\allowbreak{}finance}, \texttt{behavioral\_\allowbreak{}finance}, \texttt{macro\_\allowbreak{}finance}, \texttt{financial\_\allowbreak{}econometrics}, \texttt{entrepreneurial\_\allowbreak{}finance\_\allowbreak{}private\_\allowbreak{}equity}, \texttt{other} \\ \texttt{primary\_\allowbreak{}method} & \texttt{reduced\_\allowbreak{}form\_\allowbreak{}regression}, \texttt{difference\_\allowbreak{}in\_\allowbreak{}differences}, \texttt{instrumental\_\allowbreak{}variables}, \texttt{regression\_\allowbreak{}discontinuity}, \texttt{natural\_\allowbreak{}experiment\_\allowbreak{}event\_\allowbreak{}study}, \texttt{structural\_\allowbreak{}estimation}, \texttt{calibration\_\allowbreak{}simulation}, \texttt{theoretical\_\allowbreak{}model\_\allowbreak{}analytical}, \texttt{machine\_\allowbreak{}learning\_\allowbreak{}textual}, \texttt{survey\_\allowbreak{}or\_\allowbreak{}experiment}, \texttt{other}, \texttt{none} \\ \texttt{introduction.\allowbreak{}moves[].\allowbreak{}move} & \texttt{motivation}, \texttt{tension\_\allowbreak{}or\_\allowbreak{}puzzle}, \texttt{literature\_\allowbreak{}gap}, \texttt{preview\_\allowbreak{}of\_\allowbreak{}approach}, \texttt{preview\_\allowbreak{}of\_\allowbreak{}results}, \texttt{contribution\_\allowbreak{}statement}, \texttt{mechanism\_\allowbreak{}or\_\allowbreak{}theory\_\allowbreak{}sketch}, \texttt{identification\_\allowbreak{}defense}, \texttt{robustness\_\allowbreak{}mention}, \texttt{roadmap}, \texttt{related\_\allowbreak{}literature\_\allowbreak{}positioning}, \texttt{policy\_\allowbreak{}or\_\allowbreak{}broader\_\allowbreak{}implication}, \texttt{other} \\ \texttt{introduction.\allowbreak{}contribution\_\allowbreak{}types[]} & \texttt{new\_\allowbreak{}evidence}, \texttt{new\_\allowbreak{}data\_\allowbreak{}or\_\allowbreak{}measure}, \texttt{new\_\allowbreak{}identification}, \texttt{new\_\allowbreak{}theory\_\allowbreak{}or\_\allowbreak{}model}, \texttt{new\_\allowbreak{}method}, \texttt{resolution\_\allowbreak{}of\_\allowbreak{}puzzle}, \texttt{replication\_\allowbreak{}or\_\allowbreak{}correction}, \texttt{synthesis} \\ \texttt{section\_\allowbreak{}roles[].\allowbreak{}role} & \texttt{introduction}, \texttt{literature}, \texttt{theory}, \texttt{data}, \texttt{empirical\_\allowbreak{}design}, \texttt{results}, \texttt{robustness}, \texttt{conclusion}, \texttt{appendix}, \texttt{other} \\ \texttt{robustness.\allowbreak{}types[]} & \texttt{alternative\_\allowbreak{}specification}, \texttt{placebo}, \texttt{subsample}, \texttt{alternative\_\allowbreak{}measure}, \texttt{alternative\_\allowbreak{}sample\_\allowbreak{}period}, \texttt{falsification}, \texttt{sensitivity\_\allowbreak{}to\_\allowbreak{}parameters}, \texttt{other} \\ \texttt{introduction.\allowbreak{}literature\_\allowbreak{}positioning\_\allowbreak{}location} & \texttt{inside\_\allowbreak{}introduction}, \texttt{separate\_\allowbreak{}section}, \texttt{both}, \texttt{none} \\ \texttt{results\_\allowbreak{}presentation.\allowbreak{}statistical\_\allowbreak{}significance\_\allowbreak{}language} & \texttt{stars\_\allowbreak{}and\_\allowbreak{}tstats}, \texttt{confidence\_\allowbreak{}intervals}, \texttt{p\_\allowbreak{}values}, \texttt{mixed}, \texttt{unclear} \\ \texttt{robustness.\allowbreak{}location} & \texttt{main\_\allowbreak{}text\_\allowbreak{}section}, \texttt{within\_\allowbreak{}results}, \texttt{appendix}, \texttt{internet\_\allowbreak{}appendix}, \texttt{mixed}, \texttt{none\_\allowbreak{}seen} \\ \texttt{tables.\allowbreak{}standard\_\allowbreak{}errors\_\allowbreak{}description} & \texttt{clustered}, \texttt{robust}, \texttt{bootstrap}, \texttt{newey\_\allowbreak{}west}, \texttt{mixed}, \texttt{unclear}, \texttt{none} \\ \texttt{figures.\allowbreak{}uncertainty\_\allowbreak{}shown} & \texttt{yes}, \texttt{no}, \texttt{mixed}, \texttt{unclear}, \texttt{no\_\allowbreak{}figures} \\ \texttt{hedging.\allowbreak{}level} & \texttt{low}, \texttt{moderate}, \texttt{high} \\ \texttt{claim\_\allowbreak{}evidence[].\allowbreak{}evidence\_\allowbreak{}type} & \texttt{table}, \texttt{figure}, \texttt{equation\_\allowbreak{}or\_\allowbreak{}proof}, \texttt{prior\_\allowbreak{}literature}, \texttt{descriptive\_\allowbreak{}statistic}, \texttt{none\_\allowbreak{}stated} \\ \bottomrule\end{tabularx} \end{table} \begin{table}[htbp] \centering \small \caption{Mapping of the twelve annotation families of the mandate to the fields of the ontology (version 1.0).} \label{tab:ontology-map} \begin{tabularx}{\linewidth}{>{\raggedright\arraybackslash}p{0.26\linewidth}X} \toprule Mandate family & Schema fields \\ \midrule 1. Abstract & \texttt{abstract\_\allowbreak{}moves.\allowbreak{}\{question, gap, method, results, magnitude, contribution\}}, each with \texttt{present} and \texttt{sentence\_\allowbreak{}index} \\ 2. Introduction & \texttt{introduction.\allowbreak{}moves[]} (paragraph index, move), \texttt{contribution\_\allowbreak{}paragraph\_\allowbreak{}index}, \texttt{contribution\_\allowbreak{}types[]}, \texttt{roadmap\_\allowbreak{}present} \\ 3. Literature & \texttt{introduction.\allowbreak{}literature\_\allowbreak{}positioning\_\allowbreak{}location}; move \texttt{related\_\allowbreak{}literature\_\allowbreak{}positioning} \\ 4. Theory & \texttt{genre}; section role \texttt{theory}; move \texttt{mechanism\_\allowbreak{}or\_\allowbreak{}theory\_\allowbreak{}sketch}; evidence type \texttt{equation\_\allowbreak{}or\_\allowbreak{}proof} \\ 5. Data & section role \texttt{data}; \texttt{primary\_\allowbreak{}method}; \texttt{subfield} \\ 6. Identification & \texttt{identification.\allowbreak{}\{estimand, strategy, key\_\allowbreak{}assumptions[], threats\_\allowbreak{}addressed[], causal\_\allowbreak{}language\_\allowbreak{}strength, uses\_\allowbreak{}causal\_\allowbreak{}claim\}} \\ 7. Empirical design & section role \texttt{empirical\_\allowbreak{}design}; \texttt{primary\_\allowbreak{}method}; \texttt{tables.\allowbreak{}standard\_\allowbreak{}errors\_\allowbreak{}description} \\ 8. Results & \texttt{results\_\allowbreak{}presentation.\allowbreak{}\{economic\_\allowbreak{}significance\_\allowbreak{}discussed, statistical\_\allowbreak{}significance\_\allowbreak{}language, magnitude\_\allowbreak{}statements\_\allowbreak{}in\_\allowbreak{}intro\}} \\ 9. Robustness & \texttt{robustness.\allowbreak{}\{location, types[]\}}; move \texttt{robustness\_\allowbreak{}mention} \\ 10. Tables & \texttt{tables.\allowbreak{}\{convention\_\allowbreak{}notes, units\_\allowbreak{}stated\_\allowbreak{}in\_\allowbreak{}notes, standard\_\allowbreak{}errors\_\allowbreak{}description\}} \\ 11. Figures & \texttt{figures.\allowbreak{}uncertainty\_\allowbreak{}shown} \\ 12. Conclusion & \texttt{conclusion.\allowbreak{}\{bounded\_\allowbreak{}claims, policy\_\allowbreak{}implications, open\_\allowbreak{}questions, restates\_\allowbreak{}contribution\}} \\ Moves, claims, hedging & \texttt{introduction.\allowbreak{}moves[]}, \texttt{claim\_\allowbreak{}evidence[]} (claim, evidence type, evidence reference), \texttt{hedging.\allowbreak{}\{level, typical\_\allowbreak{}devices[]\}}, \texttt{structural\_\allowbreak{}summary}, \texttt{writing\_\allowbreak{}profile\_\allowbreak{}keywords[]}, \texttt{confidence} \\ \bottomrule \end{tabularx} \end{table} \subsection{Annotation procedure} \label{sec:extraction-procedure} Each document is annotated by one call to the Claude API (\texttt{claude-opus-5-5}, the annotation model recorded in \texttt{config/\allowbreak{}models.\allowbreak{}yaml}, with reasoning effort \texttt{medium} and a 4,000-token output limit) using prompt \texttt{document\_\allowbreak{}annotation} version 1.0 (Appendix~\ref{app:prompts}). The system prompt consists of the prompt body, the list of allowed values for every enumerated field, and the JSON schema itself; it is marked for prompt caching so that the per-document cost is dominated by the payload. The user message is the bounded payload of Section~\ref{sec:corpus-rights}, serialised as JSON. The API's constrained-decoding option for structured outputs was tried first and rejected the full schema as too large to compile into a grammar; the model is therefore asked in the prompt to return a single JSON object, and the response is parsed and validated locally against the strict schema (JSON Schema draft 2020-12, enumerations enforced, no additional properties, every listed property required). A response that fails validation is stored with its validation errors and marked invalid; invalid documents are re-run with a fresh call rather than repaired by hand, and documents that remain invalid are excluded from the rhetoric profiles and from the annotation units of the index. The record written for each document holds the annotation, the prompt and ontology versions, the model identifier returned by the API, the effort, the timestamp, the source document hash, the parser version, the payload size, token usage, cost, the API request identifier, and the code version. In the full run, all \nAnnotated{} included documents were annotated and \nAnnValid{} validate against the strict schema after re-runs; roughly one first-pass response in ten failed strict validation (typically an enumeration value outside the vocabulary or a missing property) and was re-run, at a separately logged retry cost of \apiSpendAnnotationretry{}~USD. The measured cost is \annCostPerDoc{}~USD per valid document (\annCostUSD{}~USD in total). Every count quoted from the annotations in this paper is printed from macros regenerated from the recorded artefacts. \subsection{Deterministic checks and model--rule agreement} \label{sec:extraction-checks} Independently of the model, a set of rule-based signals is computed from the same payload for every document (\texttt{deterministic\_\allowbreak{}checks} in \texttt{rhetoric.\allowbreak{}py}): the presence of a roadmap paragraph (regular expression on ``the remainder of the paper'', ``Section~\textit{n} presents'', and similar phrasings), of contribution language in the introduction (``contribute'', ``this paper is the first'', ``this paper adds''), and of a magnitude in the abstract (a number followed by a percentage, basis-point, or standard-deviation unit); the number of sentences in the abstract and of paragraphs and words in the introduction; counts of causal and hedging words in the abstract; whether table notes mention significance stars or clustered standard errors; whether the introduction or the headings mention an Internet Appendix; and a rule-based role for every section derived from its heading (\texttt{segmentation.\allowbreak{}py}). These signals serve two purposes. They enter the profiles alongside the model's judgements, so that a reader can compare, for instance, the share of introductions with a roadmap according to the model and according to the regular expression. And they provide a check on the model: for each annotated document we compute the agreement between the model's section roles and the rule-based roles over sections where the rule assigns a role, the agreement between the model's roadmap flag and the regular expression, and the agreement between the model's identification of a contribution paragraph and the contribution regular expression. Over the \nAnnValid{} valid annotations, the median per-document agreement on section roles is \roleAgreementMedian{}, the roadmap flags agree for \roadmapAgree\% of documents, and the contribution flags agree for \contribAgree\% of documents. The lower contribution agreement is expected: the regular expression requires explicit contribution vocabulary, whereas the model also recognises contribution paragraphs phrased as ``we show'' or ``our results'', and the disagreements are informative about the limits of lexical rules rather than necessarily about the model. \subsection{What the agreement figures do and do not mean} \label{sec:extraction-honesty} We state the limits of this evidence plainly. The annotations are produced by one language model under one prompt; the agreement figures above compare that model with lexical rules, and the self-consistency of the profiles compares the model with itself. Neither is inter-rater reliability in the sense used in content analysis, which requires at least two independent human coders and a chance-corrected statistic such as Krippendorff's $\alpha$ \citep{krippendorff2019content}. A human audit protocol (\texttt{experiments/\allowbreak{}protocols/\allowbreak{}}) has been prepared for the day qualified coders are available, and the rhetoric profiles should be read, until then, as a model's structured reading of the corpus whose aggregate patterns are checked against simple rules, not as a validated coding of it. The same caution applies to the downstream uses of the annotations: structured retrieval units and profile summaries inherit whatever systematic bias the annotator has, and the experiment of Section~\ref{sec:design} evaluates the usefulness of the resulting engine, not the correctness of the annotations.