F4-conf-04 — Verifying a reference list against the cited source material
Scientific question
A draft manuscript cites five open-access studies; for each, the reference list gives a bibliographic entry (authors, year, title, arXiv number) and the manuscript attributes to it a specific finding ("cited_for"). For each reference, check against the provided source extract whether (i) the title matches the document, (ii) the year is consistent with the document's submission date(s), and (iii) the attributed finding is actually stated in, or follows arithmetically from, the extract. A reference is unverifiable when the source material and the entry disagree on the title or year, or when the attributed finding is not supported by the source. The estimand is the set of unverifiable references, together with a per-reference record of the three checks and a locator for each finding check.
Inputs
data/reference_list.json— five references: {"id", "entry" (bibliographic string), "cited_for" (the attributed finding), "source_file" (the extract to check against)}.data/sources/arxiv-2609.27732v1.txt— arXiv:2609.27732v1, Armendariz (2026), CC BY 4.0: abstract, Section 6 (Tables 6, 7), Table A19.data/sources/arxiv-2606.14621v1.txt— arXiv:2606.14621v1, Li, Pearson-Merkowitz & Yokum (2026), CC BY 4.0: abstract, Section 3 (Table 2), Table 3.data/sources/arxiv-2603.03260v2.txt— arXiv:2603.03260v2, Liu & Zhao (2026; v1 March 2026, v2 April 2026), CC BY 4.0: abstract, Section 3 (Tables 2, 3), Section 5.data/sources/arxiv-2505.04555v3.txt— arXiv:2505.04555v3, Kanayama, Miyaji & Otani (v1 May 2025, v3 March 2026), CC BY 4.0: abstract, Appendix A.1 (Table 2).data/sources/arxiv-2510.03658v1.txt— arXiv:2510.03658v1, Tossou (October 2025), CC BY 4.0: abstract, Section 4.1, Table 3.- Each extract begins with a provenance header giving the title, authors, arXiv identifier with version and submission dates (the "Bibliographic record" line); the body reproduces the document's own title line, abstract and the cited sections.
data/README.mdgives retrieval times, licences and SHA-256 values.
Information-availability rules
- The extracts (header and body) are the only admissible evidence; do not use remembered content of the papers, their other versions, or external bibliographic databases.
- Year check: an entry's year is consistent if it equals the year of the first arXiv submission or the year of the version reproduced in the extract (both given in the "Bibliographic record" line). Title check: minor typographic differences (capitalisation, punctuation, diacritics, "and" vs "&") are acceptable; a different title is not. Author formatting differences (initials, order within the printed order) are not grounds for flagging.
- Finding check: every element of "cited_for" (magnitudes, uncertainty statements, timing, population) must be stated in or follow arithmetically from the extract.
- A locator names the source id and the table (row/column) or section that decides the finding check.
Required outputs
out/results.jsonwith the keys:
| key | type | unit | meaning |
|---|---|---|---|
reference_checks |
object | — | {"R1": {"title_matches": true/false, "year_matches": true/false, "finding_supported": true/false, "locator": "<source id |
unverifiable_references |
array of strings | — | ids of the references that fail at least one check (empty list if none) |
n_verified |
integer | — | number of references passing all three checks |
task_id |
string | — | "F4-conf-04" |
feasibility |
string | — | "complete", "partial" or "infeasible" — refers to the task as a whole |
feasibility_reason |
string or null | — | explanation if not complete |
data_issues |
array of {"issue","handling"} | — | problems detected in the inputs (rendering defects, internal inconsistencies of a source, etc.) and how they were handled; empty list if none |
n_input_rows |
object | lines / items | {file name: number of lines} for every file under data/ that you read (text files: number of newline-terminated lines as reported by wc -l; JSON files: number of top-level items) |
n_analysed |
integer or null | — | number of reported items (values, assertions, effects or references) you actually assessed |
random_seed |
integer or null | — | null (no randomness is involved) |
out/report.mdwith the sections: Question; Data and exclusions; Method; Results; Uncertainty; Limitations and open issues.out/analysis.py— running~/wp29/venv/bin/python out/analysis.pyfrom the workspace root must regenerateout/results.jsonfromdata/only, deterministically (the script must print the header lines and table cells it relies on).out/claims.json— a list of the central claims of the report: [{"id": "C1", "text": "...", "results_keys": ["..."], "status": "supported"|"unsupported"|"not_evaluated", "caveat": "..."}].
Constraints
- No network access. Python:
~/wp29/venv/bin/python(numpy, pandas, scipy, statsmodels available; the standard library is sufficient for this task). Do not modifydata/. - The source extracts under
data/sources/are the only admissible evidence. Do not rely on memory of the papers or on other versions of them; if a quantity is not in the extract, it is not available. - Report numbers exactly as printed in the source (same sign, same number of decimals); do not round or rescale printed values unless the key definition asks for a transformation.
- This task is designed to be completed in well under 20 minutes of work.