Data card draft
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
3 changed files +242 −5
added
docs/data_governance/DATA_CARD.md
+54 −0
@@ -0,0 +1,54 @@ | ||
| 1 | +# Data card — HTR corpus and HTR-Benchmark (draft, 2026-11-09) | |
| 2 | + | |
| 3 | +## Summary | |
| 4 | +Paired data: **source** = a passage rewritten by a large language model (Claude Opus 5.5 primary; Sonnet 5.5 / Haiku 5.5 | |
| 5 | +held-out), **target** = the original English passage from a document published before 2018-01-01 whose licence permits | |
| 6 | +the operations we perform. Both sides are stored in *placeholder form* (citations ⟦Cn⟧, math ⟦Mn⟧, links ⟦Un⟧; see | |
| 7 | +`src/htr/parse/protect.py`, protocol version recorded per pair). Metadata per pair: discipline, publication year/era, | |
| 8 | +section role, source archive, licence code, venue, author group, rewriting model, prompt version, protocol version. | |
| 9 | + | |
| 10 | +## Sources (all acquired through official channels, rate-limited, rights recorded per document) | |
| 11 | +| Source | Access route | Date evidence | Licence evidence | Scale observed | | |
| 12 | +|---|---|---|---|---| | |
| 13 | +| PMC Open Access Subset | AWS Open Data bucket `pmc-oa-opendata` (per-article JSON + JATS XML; MD5 verified) | JATS `<pub-date>` (ppub → collection → epub) | JSON `license_code` cross-checked with JATS `<license xlink:href>` | pilot 1 180 docs; Medium ingestion ~10 000 | | |
| 14 | +| arXiv (q-fin, econ; others possible) | OAI-PMH `arXivRaw` metadata + e-print source of the last pre-2018 version (3 s between requests) | v1 submission date | OAI `license` URI (only CC licences kept; the arXiv non-exclusive licence is excluded) | 162 eligible q-fin papers 2008–2017 | | |
| 15 | +| CC BY economics & finance journals (16 verified: MDPI Econometrics, JRFM, IJFS, Risks, Economies, Games; SpringerOpen Financial Innovation, J. Economic Structures, IZA journals, LAER, SJES, JIE) | Crossref REST (polite pool) + publisher PDF via Crossref text-mining links | Crossref `published-print` → `published-online` → `issued` | Crossref `license[]` (content-version vor) | 1 730 eligible articles | | |
| 16 | +| Project Gutenberg (reference / scholarly non-fiction, LoCC Q R T H B G L Z) | catalog feed + official mirror `gutenberg.pglaf.org` | front matter year (imprint / copyright / roman numerals); else author death year as an **upper bound** (flagged) | public domain in the US (catalog + licence header) | pool 7 799; 500 requested | | |
| 17 | +| NCBI Bookshelf open-access books | FTP `pub/litarch` (BITS XML) | BITS `<pub-date>` | BITS `<permissions>` (CC BY / public domain NIH works; UNKNOWN rejected) | 3 941 pre-2018 books; 300 requested | | |
| 18 | +| OpenStax / CNX pre-2018 editions | Internet Archive items; date and CC statement read **inside the PDF** | PDF copyright page | PDF licence statement | 5 trial items | | |
| 19 | + | |
| 20 | +Excluded by policy: `TDM` author manuscripts, `CC-BY-ND` / `CC-BY-NC-ND`, `UNKNOWN`/`OTHER-OA`, the arXiv non-exclusive | |
| 21 | +licence, retracted articles, non-English text, near-duplicates (MinHash Jaccard ≥ 0.8), references/acknowledgements/ | |
| 22 | +funding/ethics sections, captions and list items (kept in the parsed store, never eligible). | |
| 23 | + | |
| 24 | +## Processing | |
| 25 | +JATS/BITS, LaTeX, PDF and plain-text parsers → paragraphs with offsets into the normalised document text → eligibility | |
| 26 | +(60–400 words, ≥ 2 sentences, English ≥ 0.9, quality ≥ 0.6) → protected-span placeholders → LLM rewrite (variants A–E, | |
| 27 | +prompts `rw-1.0`) → automated validation (`val-1.0`: language, length ratio, near-copy, placeholder multiset, numbers, | |
| 28 | +entities, negation/hedging deltas, structure for D, formatting artefacts, MiniLM semantic similarity) → quarantine of | |
| 29 | +failures (kept, never deleted) → leakage-safe splits (document + near-duplicate cluster + author group; held-out discipline | |
| 30 | +and period; pilot assignments frozen). | |
| 31 | + | |
| 32 | +## Release policy (HTR-Benchmark) | |
| 33 | +- **Public subset**: pairs whose document licence is `CC0`, `PUBLIC-DOMAIN` or `CC-BY` — redistributed with attribution | |
| 34 | + (title, authors, venue, DOI/PMCID/arXiv id/Gutenberg id, licence URI, "modified: segmented and placeholder-protected"). | |
| 35 | +- **Research subset** (`CC-BY-SA`, `CC-BY-NC`, `CC-BY-NC-SA`): metadata + identifiers + scripts to rebuild locally; text | |
| 36 | + shared only under the matching licence terms (SA → CC BY-SA; NC → non-commercial research). | |
| 37 | +- **Never released**: documents of tiers C/X, quarantined pairs' rewrites of non-releasable originals, provider request | |
| 38 | + payloads beyond what the schema records. | |
| 39 | +- LLM rewrites are our outputs under Anthropic's commercial terms; their redistribution and use for training are recorded as a | |
| 40 | + compliance item to confirm before publication (`docs/RISK_REGISTER.md`). | |
| 41 | +- Takedown: a retraction or licence change at the source triggers removal from the next release and a dated note in | |
| 42 | + `docs/DECISION_LOG.md`. | |
| 43 | + | |
| 44 | +## Known biases and limitations | |
| 45 | +- Biomedicine dominates PMC; economics/finance, mathematics, physics and books are deliberate strata but smaller. | |
| 46 | +- Discipline labels are heuristic (journal title / subject categories / arXiv categories / LoCC classes). | |
| 47 | +- Pre-1950 PMC material is OCR and rarely passes quality; historical variation comes mostly from books, whose dates may | |
| 48 | + be upper bounds. | |
| 49 | +- Validation heuristics: spelled-out numbers count as missing; entity matching is a proxy. | |
| 50 | +- The rewrites reflect a specific model generation (Claude 5.5 family, late 2026) and six prompt variants. | |
| 51 | + | |
| 52 | +## Citation and acknowledgement | |
| 53 | +Acknowledge NLM/NCBI (PMC Article Datasets on AWS, accessed 2026-11), arXiv, Crossref, the publishers of the CC BY | |
| 54 | +journals, Project Gutenberg, the Internet Archive and OpenStax (Rice University). See `LICENSE_MATRIX.md`. | |
modified
paper/appendix/appendix.tex
+188 −5
@@ -1,7 +1,190 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +% Appendices | |
| 3 | +% ============================================================================ | |
| 4 | +\providecommand{\evid}[1]{\texttt{\detokenize{#1}}} | |
| 5 | +\newenvironment{promptblock}{\begin{quote}\small\ttfamily\raggedright}{\end{quote}} | |
| 6 | + | |
| 1 | 7 | \appendix |
| 2 | −\section{Rewrite prompts} | |
| 3 | −\pending{prompts rw-1.0 verbatim} | |
| 8 | + | |
| 9 | +\section{Rewrite Prompts (version \texttt{rw-1.0})} | |
| 10 | +\label{app:prompts} | |
| 11 | + | |
| 12 | +The prompts below are reproduced verbatim from \evid{src/htr/rewrite/prompts.py}. Two typographic substitutions | |
| 13 | +are made for this document only. (i) In the code, placeholders are delimited by the mathematical white square | |
| 14 | +brackets U+27E6 and U+27E7 (for example, the citation placeholder is the letter C and the index 1 between these two | |
| 15 | +glyphs); here they are written \texttt{[[C1]]}, \texttt{[[M1]]} and \texttt{[[U1]]}. (ii) LaTeX special characters | |
| 16 | +(\texttt{\%}, \texttt{\$}) are escaped. Line breaks inside a string are not significant. | |
| 17 | + | |
| 18 | +\paragraph{System message (all variants).} | |
| 19 | +\begin{promptblock} | |
| 20 | +You rewrite passages of scientific writing for a research study on how language models transform prose. You never | |
| 21 | +invent information. Placeholders of the form [[C1]], [[M1]] and [[U1]] stand for citations, equations and links; | |
| 22 | +copy each of them verbatim, exactly once per occurrence in the source, in the position where it belongs. | |
| 23 | +\end{promptblock} | |
| 24 | + | |
| 25 | +\paragraph{User message template.} The user message is the variant instruction, a blank line, the preservation | |
| 26 | +block, a blank line, and the passage in placeholder form: | |
| 27 | +\begin{promptblock} | |
| 28 | +\{instruction\}\\[0.5ex] | |
| 29 | +\{PRESERVE\}\\[0.5ex] | |
| 30 | +<passage>\\ | |
| 31 | +\{passage\_text\}\\ | |
| 32 | +</passage> | |
| 33 | +\end{promptblock} | |
| 34 | + | |
| 35 | +\paragraph{Preservation block (\texttt{PRESERVE}, all variants).} | |
| 36 | +\begin{promptblock} | |
| 37 | +Preserve every scientific proposition exactly: all numbers, units, statistical values, named entities, species, | |
| 38 | +genes, drugs, places, dates, citation markers (e.g. [12], (Smith et al., 2015), [[C1]]), equations and math tokens | |
| 39 | +(e.g. \$x\$, [[M1]]), URLs/DOIs ([[U1]]), negations, hedges and expressions of uncertainty. Do not add, remove or | |
| 40 | +reorder facts. Do not add commentary, headings, bullet points or notes. Keep the passage as a single paragraph in | |
| 41 | +English. Output only the rewritten passage. | |
| 42 | +\end{promptblock} | |
| 43 | + | |
| 44 | +\paragraph{Variant A (neutral paraphrase).} | |
| 45 | +\begin{promptblock} | |
| 46 | +Paraphrase the following passage in neutral academic English. Change the wording and sentence structure freely | |
| 47 | +while keeping the meaning identical. | |
| 48 | +\end{promptblock} | |
| 49 | + | |
| 50 | +\paragraph{Variant B (polished, standardised).} | |
| 51 | +\begin{promptblock} | |
| 52 | +Rewrite the following passage as polished, standardized contemporary academic prose: improve clarity, flow and | |
| 53 | +concision, use conventional modern scientific phrasing, and correct awkward constructions. | |
| 54 | +\end{promptblock} | |
| 55 | + | |
| 56 | +\paragraph{Variant C (concise).} | |
| 57 | +\begin{promptblock} | |
| 58 | +Rewrite the following passage more concisely (aim for about 70\% of the original length) without dropping any fact, | |
| 59 | +number, citation or qualification. | |
| 60 | +\end{promptblock} | |
| 61 | + | |
| 62 | +\paragraph{Variant D (structure-preserving).} | |
| 63 | +\begin{promptblock} | |
| 64 | +Rewrite the following passage sentence by sentence: keep the same number of sentences and the same order of ideas, | |
| 65 | +but rephrase each sentence in your own words. | |
| 66 | +\end{promptblock} | |
| 67 | + | |
| 68 | +\paragraph{Variant E (diverse sampling; two independent samples per passage).} | |
| 69 | +\begin{promptblock} | |
| 70 | +Rewrite the following passage in your own words as a fluent academic paragraph. Vary the wording naturally. | |
| 71 | +\end{promptblock} | |
| 72 | + | |
| 73 | +\paragraph{Held-out prompt \texttt{H1} (plain-language expert; never used in training).} | |
| 74 | +\begin{promptblock} | |
| 75 | +Rewrite the following passage for an expert reader in plain, direct language with short sentences. | |
| 76 | +\end{promptblock} | |
| 77 | + | |
| 78 | +\paragraph{Held-out prompt \texttt{H2} (editorial polish; never used in training).} | |
| 79 | +\begin{promptblock} | |
| 80 | +You are a copy-editor at a scientific journal. Edit the following passage for publication: fix style, tighten the | |
| 81 | +prose, keep the author's meaning. | |
| 82 | +\end{promptblock} | |
| 83 | + | |
| 84 | +\paragraph{Request settings.} Model \texttt{claude-opus-5-5} (held-out models \texttt{claude-sonnet-5-5}, | |
| 85 | +\texttt{claude-haiku-5-5}); effort \texttt{low}; \texttt{max\_tokens} 1\,500; no temperature set; thinking at the | |
| 86 | +provider default; Message Batches; samples per variant A--D: 1, E: 2; pilot budget ceiling 30~USD | |
| 87 | +(\evid{configs/rewriting/pilot.json}). A fingerprint of the prompt set (SHA-256 over version, system message, | |
| 88 | +preservation block and variants A--E) is logged with every job. | |
| 89 | + | |
| 4 | 90 | \section{Hyper-parameters} |
| 5 | −\pending{from run manifests} | |
| 6 | −\section{Reproducibility checklist} | |
| 7 | −\pending{checklist} | |
| 91 | +\label{app:hyper} | |
| 92 | + | |
| 93 | +Table~\ref{tab:hyper} lists the training and model hyper-parameters. The ``code default'' column gives the values | |
| 94 | +in the repository at this version (\evid{src/htr/training/trainer.py}, \evid{scripts/train.py}, | |
| 95 | +\evid{src/htr/models/config.py}, \evid{src/htr/models/losses.py}); where the training script overrides a library | |
| 96 | +default, both are shown. Values used for the reported models are read from the run manifests and are pending. | |
| 97 | + | |
| 98 | +\begin{table}[ht] | |
| 99 | +\centering | |
| 100 | +\caption{Hyper-parameters. Script default / trainer default where they differ.} | |
| 101 | +\label{tab:hyper} | |
| 102 | +\small | |
| 103 | +\begin{tabularx}{\textwidth}{@{}>{\raggedright\arraybackslash}X>{\raggedright\arraybackslash}p{0.3\textwidth}l@{}} | |
| 104 | +\toprule | |
| 105 | +Hyper-parameter & Code default & Reported models \\ | |
| 106 | +\midrule | |
| 107 | +\multicolumn{3}{@{}l}{\emph{Optimisation}} \\ | |
| 108 | +Optimiser & AdamW, $\beta = (0.9, 0.98)$, $\epsilon = 10^{-8}$ & \pending{manifest} \\ | |
| 109 | +Peak learning rate & $3 \times 10^{-4}$ & \pending{manifest} \\ | |
| 110 | +Weight decay & 0.01 (not on biases, norms) & \pending{manifest} \\ | |
| 111 | +Warm-up steps & 300 / 500 & \pending{manifest} \\ | |
| 112 | +Maximum steps & 5\,000 / 20\,000 & \pending{manifest} \\ | |
| 113 | +Schedule & linear warm-up, cosine to 10\% of peak & \pending{manifest} \\ | |
| 114 | +Gradient clipping (norm) & 1.0 & \pending{manifest} \\ | |
| 115 | +Gradient accumulation & 1 & \pending{manifest} \\ | |
| 116 | +Batch budget & 8\,192 tokens, at most 48 pairs & \pending{manifest} \\ | |
| 117 | +Evaluation interval & 250 / 500 steps & \pending{manifest} \\ | |
| 118 | +Early-stopping patience & 6 evaluations (validation token CE) & \pending{manifest} \\ | |
| 119 | +Precision & bf16 on CUDA, fp32 on MPS & \pending{manifest} \\ | |
| 120 | +Seeds & 1, 2, 3 & \pending{manifest} \\ | |
| 121 | +\midrule | |
| 122 | +\multicolumn{3}{@{}l}{\emph{Model}} \\ | |
| 123 | +Size design point & tiny (Table~\ref{tab:sizes}) & \pending{manifest} \\ | |
| 124 | +Dropout & 0.1 & \pending{manifest} \\ | |
| 125 | +Positional encoding & RoPE (self-attention) & \pending{manifest} \\ | |
| 126 | +Maximum source / target length & 512 / 512 tokens & \pending{manifest} \\ | |
| 127 | +Tokenizer & SentencePiece unigram, train split only; design vocabulary 32\,000 & \pending{vocabulary size} \\ | |
| 128 | +Copy gate & on & \pending{manifest} \\ | |
| 129 | +Style tokens $T_s$ (M3) & 4 & \pending{manifest} \\ | |
| 130 | +Style classes $K$ (M3) & (discipline, era) buckets with $\geq 20$ training pairs, plus ``other'' & \pending{$K$} \\ | |
| 131 | +Gradient-reversal $\lambda$ (M3) & 0.1 & \pending{manifest} \\ | |
| 132 | +\midrule | |
| 133 | +\multicolumn{3}{@{}l}{\emph{Loss}} \\ | |
| 134 | +Label smoothing $\varepsilon$ & 0.1 (0 in overfit test and validation) & \pending{manifest} \\ | |
| 135 | +Contrastive temperature $\tau$ & 0.07 & \pending{manifest} \\ | |
| 136 | +Loss weights & Table~\ref{tab:loss-weights} & \pending{manifest} \\ | |
| 137 | +\midrule | |
| 138 | +\multicolumn{3}{@{}l}{\emph{Decoding and evaluation}} \\ | |
| 139 | +Decoding & greedy (beam width chosen on validation) & \pending{freeze record} \\ | |
| 140 | +Bootstrap replicates & 1\,000 (development) / 10\,000 (confirmatory) & \pending{freeze record} \\ | |
| 141 | +\bottomrule | |
| 142 | +\end{tabularx} | |
| 143 | +\end{table} | |
| 144 | + | |
| 145 | +\section{Reproducibility Checklist} | |
| 146 | +\label{app:checklist} | |
| 147 | + | |
| 148 | +The checklist follows the structure of the ACL Responsible NLP Research checklist. ``Yes'' means the item exists in | |
| 149 | +the repository at this version; ``Partial'' that it exists for the completed gates only; ``Pending'' that it | |
| 150 | +depends on runs not yet made. | |
| 151 | + | |
| 152 | +\begin{center} | |
| 153 | +\small | |
| 154 | +\begin{tabularx}{\textwidth}{@{}>{\raggedright\arraybackslash}X>{\raggedright\arraybackslash}p{0.1\textwidth}>{\raggedright\arraybackslash}p{0.33\textwidth}@{}} | |
| 155 | +\toprule | |
| 156 | +Item & Status & Where \\ | |
| 157 | +\midrule | |
| 158 | +\multicolumn{3}{@{}l}{\emph{A. Claims, limitations and risks}} \\ | |
| 159 | +Main claims stated and scoped (no first-inversion, authorship or detection claim) & Yes & Sections~\ref{sec:intro}, \ref{sec:related} \\ | |
| 160 | +Limitations discussed & Pending & Limitations section (after results) \\ | |
| 161 | +Potential risks and misuse discussed & Yes & Section~\ref{sec:ethics} \\ | |
| 162 | +\midrule | |
| 163 | +\multicolumn{3}{@{}l}{\emph{B. Scientific artefacts}} \\ | |
| 164 | +Sources of data cited and described & Yes & Section~\ref{sec:data} \\ | |
| 165 | +Licence of every document recorded and respected & Yes & licence policy, per-document metadata \\ | |
| 166 | +Intended use of artefacts stated & Partial & Section~\ref{sec:ethics}; model card template \\ | |
| 167 | +Personal or offensive content checked & Partial & published scholarly text only; no systematic screen \\ | |
| 168 | +Documentation of artefacts (data card, model card) & Pending & model-card template exists \\ | |
| 169 | +Dataset statistics (counts, splits) reported & Partial & pilot corpus and splits; Medium stage \pending{counts} \\ | |
| 170 | +\midrule | |
| 171 | +\multicolumn{3}{@{}l}{\emph{C. Computational experiments}} \\ | |
| 172 | +Parameter counts, compute budget, infrastructure & Partial & design points (Table~\ref{tab:sizes}); per-run counts \pending{manifests} \\ | |
| 173 | +Experimental setup and hyper-parameters & Partial & Section~\ref{sec:design}, Table~\ref{tab:hyper} \\ | |
| 174 | +Descriptive statistics with error bars over seeds & Pending & 3 seeds, clustered bootstrap planned \\ | |
| 175 | +Packages and versions for pre-processing and evaluation & Yes & \evid{pyproject.toml}; versions logged per run \\ | |
| 176 | +Code for data acquisition, training and evaluation & Yes & \evid{src/htr/}, \evid{scripts/} \\ | |
| 177 | +Random seeds, data hashes, run manifests & Yes & per-run manifests; split seed 20261109 \\ | |
| 178 | +Pre-registration of confirmatory tests & Partial & draft v0.1; freeze \pending{commit} \\ | |
| 179 | +\midrule | |
| 180 | +\multicolumn{3}{@{}l}{\emph{D. Human annotators and participants}} \\ | |
| 181 | +Instructions, recruitment, payment, consent & Pending & protocol in Section~\ref{sec:design} \\ | |
| 182 | +Ethics review & Pending & UQO \pending{approval} \\ | |
| 183 | +\midrule | |
| 184 | +\multicolumn{3}{@{}l}{\emph{E. Use of AI assistants and paid APIs}} \\ | |
| 185 | +Model identifiers, prompts and settings of the rewriting LLM & Yes & Section~\ref{sec:rewrite}, Appendix~\ref{app:prompts} \\ | |
| 186 | +API spend disclosed & Yes & 29.68 USD (Table~\ref{tab:pilot-jobs}) \\ | |
| 187 | +Use of AI assistance in research and writing disclosed & Yes & Section~\ref{sec:ethics} \\ | |
| 188 | +\bottomrule | |
| 189 | +\end{tabularx} | |
| 190 | +\end{center} | |
modified
paper/main.pdf
+0 −0
Binary file not shown.