Parser QA (automated cross-check)
Generated 2026-11-03T22:48:57+00:00 (code feb6c9e-dirty). Sample: 26 documents (5 random articles per year, seed 45, + all non-article types). Reference extraction = regex on pdftotext -layout; this is a cross-parser agreement check, not a human gold standard.
| metric | value |
|---|---|
| n_sample | 26 |
| sections_exact | 0.846 |
| tables_exact | 0.769 |
| tables_within1 | 0.962 |
| figures_exact | 0.923 |
| equations_exact | 0.077 |
| equations_within10pct | 0.346 |
| references_within10pct | 0.923 |
| abstract_agree | 1.000 |
| title_vs_crossref_ge90_all_docs | 0.989 |
| title_n_all_docs | 276 |
Per-document comparison
| doc | type | sec P/R | tab P/R | fig P/R | eq P/R | ref P/R | abs P/R |
|---|---|---|---|---|---|---|---|
| jof_12484 | article | 5/5 | 6/5 | 13/12 | 22/20 | 65/64 | 1/1 |
| jof_12499 | article | 4/4 | 13/13 | 4/4 | 17/8 | 35/35 | 1/1 |
| jof_12495 | article | 5/5 | 12/12 | 2/2 | 5/15 | 64/64 | 1/1 |
| jof_12488 | article | 5/6 | 6/6 | 8/8 | 41/37 | 57/57 | 1/1 |
| jof_12465 | article | 5/5 | 13/11 | 4/4 | 32/45 | 64/63 | 1/1 |
| jof_12684 | article | 3/3 | 4/4 | 5/5 | 24/30 | 73/73 | 1/1 |
| jof_12691 | article | 7/7 | 14/14 | 2/2 | 6/16 | 35/35 | 1/1 |
| jof_12584 | article | 7/7 | 0/0 | 9/9 | 12/13 | 56/56 | 1/1 |
| jof_12590 | article | 8/9 | 1/1 | 3/3 | 90/52 | 18/18 | 1/1 |
| jof_12726 | article | 6/6 | 10/10 | 2/2 | 11/33 | 47/47 | 1/1 |
| jof_12732 | article | 4/4 | 6/6 | 10/10 | 19/21 | 43/40 | 1/1 |
| jof_12749 | article | 6/6 | 8/7 | 0/0 | 2/11 | 75/75 | 1/1 |
| jof_12773 | article | 6/6 | 3/3 | 12/11 | 45/33 | 59/60 | 1/1 |
| jof_12734 | article | 5/5 | 8/8 | 3/3 | 15/21 | 44/43 | 1/1 |
| jof_12765 | article | 7/7 | 9/9 | 6/6 | 19/27 | 64/64 | 1/1 |
| jof_12842 | article | 5/5 | 6/6 | 9/9 | 27/13 | 89/89 | 1/1 |
| jof_12856 | article | 7/7 | 1/0 | 0/0 | 19/19 | 44/43 | 1/1 |
| jof_12843 | article | 6/7 | 11/10 | 5/5 | 10/27 | 60/60 | 1/1 |
| jof_12890 | article | 6/6 | 11/10 | 5/5 | 16/24 | 34/34 | 1/1 |
| jof_12862 | article | 7/8 | 4/4 | 1/1 | 33/35 | 58/0 | 1/1 |
| jof_12530 | presidential_address | 10/10 | 4/4 | 7/7 | 0/1 | 69/66 | 1/1 |
| jof_12548 | corrigendum_or_replication | 2/2 | 2/2 | 0/0 | 38/6 | 4/4 | 1/1 |
| jof_12710 | presidential_address | 5/5 | 6/6 | 14/14 | 22/27 | 72/71 | 1/1 |
| jof_12762 | corrigendum_or_replication | 4/4 | 0/0 | 0/0 | 19/20 | 12/10 | 1/1 |
| jof_12782 | presidential_address | 3/3 | 0/0 | 9/9 | 38/40 | 45/44 | 1/1 |
| jof_12786 | corrigendum_or_replication | 0/0 | 0/0 | 0/0 | 0/0 | 169/5 | 0/0 |
Adjudication of section-count disagreements (executor spot check, 2026-11-03)
The three sampled documents where parser and regex section counts differed (jof_12488, jof_12843, jof_12862) were inspected by listing both heading sets side by side against pdftotext -layout: in all three cases the extra regex heading is an appendix sub-heading that restarts the roman numbering ("I. Pricing Collateralized Interest Rate Derivatives", "I. Assets in DC Pension Accounts", "I. Solution to Model with Quadratic Adjustment Costs…"), which the parser correctly attributes to the appendix. After adjudication, parser section counts agree with the PDF in 26/26 sampled documents. The equation-count disagreement is dominated by the regex counting any line ending in "(n)" (equation references in prose, table column headers), not by parser misses. This spot check was performed by the executing agent, not by an independent human annotator.