Validation report — FIRC-0001, exhibit layer (protocol Part IX; Stage D laptop batches L1/L6)
Prepared 2026-11-05 by the laptop executor on the review worktree worktrees/FIRC-0001 (branch review-20261103). The node-side report (DuckDB audits, regeneration of the 29 exports by N13) is UQO-WP-REVIEW/reproducibility/FIRC-0001/validation_report_N.md; this report covers what was re-run on the laptop and the resulting verdict for the exhibit layer.
What was re-run, where, how long
| step | command | machine | wall time | log |
|---|---|---|---|---|
| exhibits | OMP_NUM_THREADS=4 nice -n 10 python3 -I code/make_tables.py |
laptop (Python 3.14.4, matplotlib 3.10.9) | ≈ 3 s | reproducibility/FIRC-0001/logs/L1_make_tables_*.log, L1_build_*.log |
| paper | latexmk -pdf -interaction=nonstopmode main.tex (pdflatex + bibtex, aer.bst) |
laptop (TeX Live 2026) | ≈ 40 s | L1_build_20261105-142858.log |
| tests | python3 -I -m pytest -q tests |
laptop (pytest 9.1.1) | ≈ 1 s | L6_pytest_*.log |
| DOI resolution | curl × 24 against Crossref / DataCite, 0.3 s apart |
laptop | ≈ 20 s | L4_crossref_20261105-141223.json |
No LLM or embedding call was made; nothing was written on any node; data/stats.json and the 29 committed CSV exports were not modified by any laptop step (the generator no longer rewrites stats.json; sha256(data/stats.json) is asserted unchanged by tests/test_tables.py::test_regeneration_reproduces_committed_tables).
What matched
- Export reconciliation (21 assertions, all pass): the marginal CSV tables sum to the headline counts of
stats.json— ledger rows 1,123,098; stored mentions 308,065 (by form, by year, by confidence); projects 121,099 (by type, status, span, mention bucket, first-seen year, sector, index membership); filings 527,548 (by form, by year × form, by year, by parse status); sections 1,342,283; universe 2,564 = 491 + 9 + 2,064 = 2,499 + 65 = Σ sectors; 10-K and 10-Q item tables sum to the per-form ledger rows and mentions; amount-bearing projects 56,500 inamount_dist.csv,projects_by_type.csvandstats.json. - Regeneration:
make_tables.py --out <scratch>reproduces every committedpaper/tables/*.texbyte for byte and the committedresults/inline_numbers.json(207 macros). Figures are regenerated as PDF (vector;CreationDatesuppressed) — compared visually, not byte-wise. - Build:
latexmkexit 0, 0 LaTeX errors, 0 undefined citations/references, bibtex 0 warnings, 39 pages. Overfull boxes: 17 (baseline 14); the three new ones are the wider Tables 3 and 5 whose notes still sit inside theadjustbox(item LITE-10, Stage E/L3 moves the notes outside), and the 835 pt box is the hand-typed regex ofappendix.tex:19that Stage E replaces by\input{tables/gate_regex}. - Prompt and gate regex:
paper/tables/prompt_verbatim.texandgate_regex.texare now generated from the frozen snapshot (code/audit/spid_snapshot/llm.py,taxonomy.py,rules.py, SHA-256 inSHA256SUMS) by parsing the source withast; the printed regex concatenates to the_PROJECT_GATEpattern ofrules.pyexactly (three source lines, 26 alternatives), and the prompt lists the 21 taxonomy labels in the orderTYPE_LABELSsends them. - Literals removed from the generators:
make_tables.py—2{,}587(nowdata/documented_inputs.json:iwv_equity_lines_reported, labelled "count reported by the build log, source file not archived"),(500 CIKs)(nowstats.json:universe.n_sp500/sp500_ciks),2010-01-04/2026-10-27(nowstats.json:filing_date_rangeor min/max offilings_by_form.csv),128.5(nowstats.json:amount_total_median_musdoramount_dist.csv),2700(y-limit nowuniverse.n × 1.05),300(label offset now 0.7 % of the maximum),170(nowdocumented_inputs.json:snippet_chars);export_stats.py— thespid_runsstrings (2026-06 (gpt-4o-mini, 16 workers, 153,894 sections),… 48 workers, 968,134 sections) are now read by regex from the two run logs and from the ledger (Nonewhen absent).tests/test_tables.py::test_make_tables_has_no_data_literalstokenises the generator and fails on any integer literal of three or more digits other than the scale100(1e3/1e6/1e9are exponent forms).
What did not match / is still open
export_stats.pywas edited but not run on the laptop (it needs the two DuckDB files on M3U96b); the new keys it exports (filing_date_range,amount_total_median_musd,amount_max,amount_ge_50b,amount_le_zero,ledger_10k_mdna_median_words,annual_panel_firms,sp500_ciks,gate_pass_total,calls_by_form,backfilled_rows,outage,conf_exact_default_count,amount_raw_deviations,fiscal_year_null_mentions, structuredspid_runs) and its three new dumps (ledger_collisions.csv,cohort_by_run.csv,calls_by_form.csv, plus the seeded bootstrapprojects_by_type_ci.csv) appear in the exhibits only once N13 has re-run it and its 29-file diff against the committed exports is empty. Until thenmake_tables.pyfalls back to the committed exports and to the N1/N3 result files (results/ledger/*.json,results/resolver/*), and the "95 % CI of the median" column of Table 5 prints--.- The figure PNGs under
paper/figures/are produced by no script (DATA-15); they are kept for the Word export only. - The release of the derived DuckDB tables (AD-1) and the licence of code and tables (AD-3) are author decisions; no
LICENSEfile is committed.
Reproducibility verdict
- Exhibit layer (tables, figures, inline numbers from
data/andresults/): R1 — reproduced from the documented analysis data; the raw filings (254.5 GB) and the two DuckDB files are too large to ship and the LLM extraction is not re-run. - Export layer (
data/*.csv,stats.jsonfrom the DuckDB files): provisional R2 until N13 reports its diff; R1 if the 29 committed files are regenerated byte-identically on M3U96b, R4 if any headline count moves. - Extraction layer (LLM calls): not reproducible by design — re-runnable and ledgered, with the determinism statement of
docs/DATA_AVAILABILITY.md; stability under re-submission is unmeasured (AR-LLM-1 not granted).