SPB Git

spb/firc0001_uqo

Public
6.1 KB

# Validation report — FIRC-0001, exhibit layer (protocol Part IX; Stage D laptop batches L1/L6)

Prepared 2026-11-05 by the laptop executor on the review worktree worktrees/FIRC-0001 (branch review-20261103). The node-side report (DuckDB audits, regeneration of the 29 exports by N13) is UQO-WP-REVIEW/reproducibility/FIRC-0001/validation_report_N.md; this report covers what was re-run on the laptop and the resulting verdict for the exhibit layer.

# What was re-run, where, how long

step command machine wall time log
exhibits OMP_NUM_THREADS=4 nice -n 10 python3 -I code/make_tables.py laptop (Python 3.14.4, matplotlib 3.10.9) ≈ 3 s reproducibility/FIRC-0001/logs/L1_make_tables_*.log, L1_build_*.log
paper latexmk -pdf -interaction=nonstopmode main.tex (pdflatex + bibtex, aer.bst) laptop (TeX Live 2026) ≈ 40 s L1_build_20261105-142858.log
tests python3 -I -m pytest -q tests laptop (pytest 9.1.1) ≈ 1 s L6_pytest_*.log
DOI resolution curl × 24 against Crossref / DataCite, 0.3 s apart laptop ≈ 20 s L4_crossref_20261105-141223.json

No LLM or embedding call was made; nothing was written on any node; data/stats.json and the 29 committed CSV exports were not modified by any laptop step (the generator no longer rewrites stats.json; sha256(data/stats.json) is asserted unchanged by tests/test_tables.py::test_regeneration_reproduces_committed_tables).

# What matched

  • Export reconciliation (21 assertions, all pass): the marginal CSV tables sum to the headline counts of stats.json — ledger rows 1,123,098; stored mentions 308,065 (by form, by year, by confidence); projects 121,099 (by type, status, span, mention bucket, first-seen year, sector, index membership); filings 527,548 (by form, by year × form, by year, by parse status); sections 1,342,283; universe 2,564 = 491 + 9 + 2,064 = 2,499 + 65 = Σ sectors; 10-K and 10-Q item tables sum to the per-form ledger rows and mentions; amount-bearing projects 56,500 in amount_dist.csv, projects_by_type.csv and stats.json.
  • Regeneration: make_tables.py --out <scratch> reproduces every committed paper/tables/*.tex byte for byte and the committed results/inline_numbers.json (207 macros). Figures are regenerated as PDF (vector; CreationDate suppressed) — compared visually, not byte-wise.
  • Build: latexmk exit 0, 0 LaTeX errors, 0 undefined citations/references, bibtex 0 warnings, 39 pages. Overfull boxes: 17 (baseline 14); the three new ones are the wider Tables 3 and 5 whose notes still sit inside the adjustbox (item LITE-10, Stage E/L3 moves the notes outside), and the 835 pt box is the hand-typed regex of appendix.tex:19 that Stage E replaces by \input{tables/gate_regex}.
  • Prompt and gate regex: paper/tables/prompt_verbatim.tex and gate_regex.tex are now generated from the frozen snapshot (code/audit/spid_snapshot/llm.py, taxonomy.py, rules.py, SHA-256 in SHA256SUMS) by parsing the source with ast; the printed regex concatenates to the _PROJECT_GATE pattern of rules.py exactly (three source lines, 26 alternatives), and the prompt lists the 21 taxonomy labels in the order TYPE_LABELS sends them.
  • Literals removed from the generators: make_tables.py — 2{,}587 (now data/documented_inputs.json:iwv_equity_lines_reported, labelled "count reported by the build log, source file not archived"), (500 CIKs) (now stats.json:universe.n_sp500 / sp500_ciks), 2010-01-04/2026-10-27 (now stats.json:filing_date_range or min/max of filings_by_form.csv), 128.5 (now stats.json:amount_total_median_musd or amount_dist.csv), 2700 (y-limit now universe.n × 1.05), 300 (label offset now 0.7 % of the maximum), 170 (now documented_inputs.json:snippet_chars); export_stats.py — the spid_runs strings (2026-06 (gpt-4o-mini, 16 workers, 153,894 sections), … 48 workers, 968,134 sections) are now read by regex from the two run logs and from the ledger (None when absent). tests/test_tables.py::test_make_tables_has_no_data_literals tokenises the generator and fails on any integer literal of three or more digits other than the scale 100 (1e3/1e6/1e9 are exponent forms).

# What did not match / is still open

  • export_stats.py was edited but not run on the laptop (it needs the two DuckDB files on M3U96b); the new keys it exports (filing_date_range, amount_total_median_musd, amount_max, amount_ge_50b, amount_le_zero, ledger_10k_mdna_median_words, annual_panel_firms, sp500_ciks, gate_pass_total, calls_by_form, backfilled_rows, outage, conf_exact_default_count, amount_raw_deviations, fiscal_year_null_mentions, structured spid_runs) and its three new dumps (ledger_collisions.csv, cohort_by_run.csv, calls_by_form.csv, plus the seeded bootstrap projects_by_type_ci.csv) appear in the exhibits only once N13 has re-run it and its 29-file diff against the committed exports is empty. Until then make_tables.py falls back to the committed exports and to the N1/N3 result files (results/ledger/*.json, results/resolver/*), and the "95 % CI of the median" column of Table 5 prints --.
  • The figure PNGs under paper/figures/ are produced by no script (DATA-15); they are kept for the Word export only.
  • The release of the derived DuckDB tables (AD-1) and the licence of code and tables (AD-3) are author decisions; no LICENSE file is committed.

# Reproducibility verdict

  • Exhibit layer (tables, figures, inline numbers from data/ and results/): R1 — reproduced from the documented analysis data; the raw filings (254.5 GB) and the two DuckDB files are too large to ship and the LLM extraction is not re-run.
  • Export layer (data/*.csv, stats.json from the DuckDB files): provisional R2 until N13 reports its diff; R1 if the 29 committed files are regenerated byte-identically on M3U96b, R4 if any headline count moves.
  • Extraction layer (LLM calls): not reproducible by design — re-runnable and ledgered, with the determinism statement of docs/DATA_AVAILABILITY.md; stability under re-submission is unmeasured (AR-LLM-1 not granted).
Files · b6ab6bf94a525e85ed76645675e7fc83ece8b176
Menu

Theme