L6: replication package — tests (22 checks: export reconciliation, byte-identical regeneration, no data literal), run_all.sh, data availability statement draft + provenance table + determinism paragraph, exhibit pins, README program-to-exhibit map, validation report (exhibit layer R1, exports provisional R2 pending N13)
Items: FIRC0001-ADVE-08, FIRC0001-EDIT-11, FIRC0001-AI_C-18, FIRC0001-ECON-17, FIRC0001-DATA-10, FIRC0001-DATA-12, FIRC0001-AI_C-14. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
8 changed files +282 −24
modified
README.md
+47 −9
@@ -1,20 +1,58 @@ | ||
| 1 | 1 | # UQO Working Paper No. 0001 (FIRC-funded) — From Corporate Disclosures to a Russell 3000 Strategic Project Database |
| 2 | 2 | |
| 3 | −*Identification, Measurement, Validation, and Research Applications* — Manel Kammoun, Djerrycharli Tandjambianda, Simon-Pierre Boucher (order to be confirmed; see `docs/contributions.md`). Version 1.0, October 28, 2026 — **working draft, research database under validation**. Research funded by the FIRC (Fonds institutionnel de recherche et de création, UQO). | |
| 3 | +*Identification, Measurement, Validation, and Research Applications* — Manel Kammoun, Djerrycharli Tandjambianda, Simon-Pierre Boucher (order to be confirmed; see `docs/contributions.md`). Version 1.0, October 28, 2026 — **working draft, research database under validation**; revision branch `review-20261103` (Stage D, November 2026). Research funded by the FIRC (Fonds institutionnel de recherche et de création, UQO). | |
| 4 | 4 | |
| 5 | −> **TL;DR** — SPID turns 527,548 SEC filings (10-K/10-Q/8-K, 2010–Oct 2026) of 2,564 issuers (S&P 500 ∪ iShares IWV Russell 3000 proxy) into 308,065 evidence-linked project mentions and 121,099 resolved candidate projects (21-category taxonomy, status, years, locations, technologies, partners, amounts). The paper documents the universe, the distributed acquisition/parsing architecture, the LLM extraction contract, the resolution rule and its failure modes, descriptive patterns, and a validation protocol — stating explicitly which gates are met and which are not. | |
| 5 | +> **TL;DR** — SPID turns 527,548 SEC filings (10-K/10-Q/8-K, 2010–Oct 2026) of 2,564 issuers (S&P 500 ∪ iShares IWV Russell 3000 proxy, 85.3 % of the index's 2,996 constituents) into 308,065 section-linked candidate project mentions and 121,099 candidate projects under the current resolution key (21-category taxonomy, status, years, locations, technologies, partners, amounts). The paper documents the universe, the distributed acquisition/parsing architecture, the LLM extraction contract (two runs under one prompt lineage), the resolution rule and its failure modes, descriptive patterns, and a validation protocol — stating explicitly which gates are met and which are not. | |
| 6 | 6 | |
| 7 | 7 | ## Layout |
| 8 | 8 | ``` |
| 9 | −paper/ main.tex, sections/, appendix/, tables/ (generated), figures/ (generated), references.bib → main.pdf | |
| 10 | −code/ export_stats.py (run on the master node against the two DuckDB files) · make_tables.py (CSV → LaTeX/figures) | |
| 11 | −data/ CSV exports + stats.json produced by export_stats.py (every number in the paper) | |
| 12 | −docs/ implementation_audit.md · universe_definition.md · project_ontology.md · data_dictionary.md · contributions.md · research_log.md | |
| 9 | +paper/ main.tex, sections/, appendix/, tables/ (generated), figures/ (generated), references.bib → main.pdf | |
| 10 | +code/ export_stats.py (master node, read-only DuckDB → data/) · make_tables.py (data/ + results/ → tables, figures, macros) | |
| 11 | +code/audit/ Stage D read-only audit scripts a01…a13 (M3U96b) and the frozen pipeline snapshot spid_snapshot/ (SHA256SUMS) | |
| 12 | +data/ CSV exports + stats.json (export_stats.py) · documented_inputs.json (the few non-query values, each with its source) | |
| 13 | + datamodel_audit.csv (Appendix C rows) · example_project.json (written by make_tables.py) · metadata/ (universe artefacts + hashes, N10) | |
| 14 | +results/ Stage D audit outputs (≤ 5 MB each, with manifest.json) · inline_numbers.json · russell_coverage.json · tables_diff.txt | |
| 15 | +docs/ implementation_audit.md · universe_definition.md · universe_coverage.md · project_ontology.md · data_dictionary.md | |
| 16 | + DATA_AVAILABILITY.md (DAS draft, provenance table, determinism paragraph) · annotation_protocol.md (N11) · contributions.md · research_log.md | |
| 17 | +tests/ test_tables.py — export reconciliation, byte-identical regeneration, no data literal in the generator | |
| 18 | +reproducibility/ validation_report.md (verdict), environment/requirements-exhibits.txt; node lock files at the root (requirements.lock, .python-version, environment.md) | |
| 19 | +run_all.sh single entry point (exhibits → tests → paper; --with-node / --with-audit for the node steps) | |
| 13 | 20 | ``` |
| 14 | −Database and code: `uqo-research-db.git` (revision 2324b58) on the UQO Git server; raw filings are not redistributed. | |
| 21 | +Database and pipeline code: `uqo-research-db.git` (revision `2324b58`) on the UQO Git server; raw filings are not redistributed (`docs/DATA_AVAILABILITY.md`). | |
| 15 | 22 | |
| 16 | 23 | ## Rebuild the exhibits |
| 17 | 24 | ```bash |
| 18 | −# on the master node: python code/export_stats.py ~/UQO_RESEARCH_DB data/ | |
| 19 | −python3 code/make_tables.py && cd paper && latexmk | |
| 25 | +./run_all.sh # laptop: python3 -I code/make_tables.py → tests → latexmk (≈ 1 min) | |
| 26 | +./run_all.sh --with-node # master node (M3U96b): export_stats.py first (read-only DuckDB, threads=4, ≈ 10 min + bootstrap) | |
| 27 | +./run_all.sh --with-audit # master node: Stage D audits (code/audit/run_all.sh) before the exhibits | |
| 28 | +python3 -I -m pytest -q tests # checks only | |
| 20 | 29 | ``` |
| 30 | +`make_tables.py` never writes `data/stats.json`; the example project of Table 9 goes to `data/example_project.json`. Inline numbers quoted in the text are LaTeX macros `\SPID<name>` generated in `paper/tables/numbers.tex` (loaded in the preamble of `main.tex`); their value, source file and formula are listed in `results/inline_numbers.json`. | |
| 31 | + | |
| 32 | +## Program-to-exhibit map | |
| 33 | + | |
| 34 | +| exhibit | generated file | script | inputs | | |
| 35 | +|---|---|---|---| | |
| 36 | +| Table 1 universe reconciliation | `paper/tables/tab_reconciliation.tex` | `make_tables.py` | `stats.json:universe`, `documented_inputs.json:iwv_equity_lines_reported` | | |
| 37 | +| Table 2 filings by form | `tab_filings.tex` | `make_tables.py` | `filings_by_form.csv`, `parse_status.csv`, `stats.json` | | |
| 38 | +| Table 3 sections and ledger (+ note) | `tab_sections.tex`, `tab_sections_note.tex` | `make_tables.py` | `sections_by_form.csv`, `ledger_by_form.csv`, `mentions_by_form.csv`, `results/ledger/table3_ledger_vs_stored.csv`, `results/ledger/ledger_collisions.csv` | | |
| 39 | +| Table 4 items in scope | `tab_items.tex` | `make_tables.py` | `ledger_10k_items.csv`, `ledger_10q_items.csv`, `results/ledger/gate_pass_by_form_item.csv` | | |
| 40 | +| Table 5 projects by type (+ note) | `tab_types.tex`, `tab_types_note.tex` | `make_tables.py` | `projects_by_type.csv`, `projects_by_type_ci.csv` (N13), `results/resolver/overmerge_by_type.csv`, `amount_dist.csv` | | |
| 41 | +| Table 6 by sector | `tab_sectors.tex` | `make_tables.py` | `projects_by_sector.csv`, `companies_by_sector.csv` | | |
| 42 | +| Table 7 status / mentions / span | `tab_status.tex` | `make_tables.py` | `projects_by_status.csv`, `mentions_per_project.csv`, `duration.csv` | | |
| 43 | +| Table 8 amounts and projects per issuer (+ note) | `tab_amounts.tex`, `tab_amounts_note.tex` | `make_tables.py` | `amount_dist.csv`, `projects_per_issuer.csv`, `stats.json:amount_*` | | |
| 44 | +| Table 9 provenance example | `tab_provenance.tex` | `make_tables.py` | `example_provenance.csv`, `documented_inputs.json:snippet_chars` | | |
| 45 | +| Table 12 data-model audit (longtable) | `tab_datamodel.tex` | `make_tables.py` | `data/datamodel_audit.csv` | | |
| 46 | +| Tables 14–17 (appendix) | `tab_coverage_year.tex`, `tab_top_issuers.tex`, `tab_confidence.tex`, `tab_graph.tex` | `make_tables.py` | `issuers_by_year.csv`, `filings_by_year_form.csv`, `mentions_by_year.csv`, `projects_by_first_seen_year.csv`, `top_issuers.csv`, `confidence_dist.csv`, `kg.csv` | | |
| 47 | +| Appendix A taxonomy | `tab_taxonomy.tex` | `make_tables.py` | `code/audit/spid_snapshot/taxonomy.py` | | |
| 48 | +| Appendix B prompt and gate regex | `prompt_verbatim.tex`, `gate_regex.tex` | `make_tables.py` | `code/audit/spid_snapshot/llm.py`, `rules.py` (read with `ast`, never retyped) | | |
| 49 | +| Stage D tables (when `results/` is present) | `tab_spec_curve.tex`, `tab_top_name_tokens.tex`, `tab_overmerge.tex`, `tab_june_october.tex`, `tab_table3_calls.tex`, `tab_grounding.tex`, `tab_focus.tex`, `tab_cohort.tex`, `tab_fp_classes.tex`, `tab_timeseries_cohort.tex`, `tab_benchmark_8k.tex`, `tab_rules_agreement.tex` | `make_tables.py` (generic CSV → tabular) | `results/<package>/*.csv` from `code/audit/a*.py` | | |
| 50 | +| Figure 1 coverage (integer year ticks) | `paper/figures/fig_coverage.pdf` | `make_tables.py` | `issuers_by_year.csv` | | |
| 51 | +| Figure 3 projects by type | `fig_types.pdf` | `make_tables.py` | `projects_by_type.csv` | | |
| 52 | +| Figure 4 confidence | `fig_confidence.pdf` | `make_tables.py` | `confidence_dist.csv` | | |
| 53 | +| Figure 5 mentions and new projects by year (y from 0; second panel per issuer; 2010 / 2026 shaded) | `fig_mentions_year.pdf` | `make_tables.py` | `mentions_by_year.csv`, `projects_by_first_seen_year.csv` | | |
| 54 | +| Figure 2 pipeline diagram | `fig_pipeline.pdf` | hand-drawn (TikZ/PDF, not generated) | — | | |
| 55 | +| inline numbers (`\SPID…`) | `paper/tables/numbers.tex`, `results/inline_numbers.json` | `make_tables.py` | all of the above; `results/**.json` (N1, N3, L5) | | |
| 56 | +| `data/*.csv`, `data/stats.json` | 29 CSV + 3 new (`ledger_collisions`, `cohort_by_run`, `calls_by_form`) + `projects_by_type_ci.csv` | `export_stats.py` (node, read-only) | `edgar_research.duckdb`, `spid.duckdb`, `data/metadata/*.json`, run logs | | |
| 57 | + | |
| 58 | +Figure numbers follow the order of `main.tex`; the `.png` copies under `paper/figures/` are not produced by any script and are kept only for the Word export (`code/tex2docx`). | |
added
docs/DATA_AVAILABILITY.md
+33 −0
@@ -0,0 +1,33 @@ | ||
| 1 | +# Data and code availability statement — draft for §10.2 (L6, cluster L; FIRC0001-DATA-12, ADVE-08, AI_C-14, ECON-17) | |
| 2 | + | |
| 3 | +Drafted 2026-11-05 (Stage D, laptop executor) following the AEA Data and Code Availability Standard (DCAS v1.0). Stage E inserts the statement in `paper/sections/conclusion.tex` §10.2; the two release decisions marked **author decision** are open (`improvements/FIRC-0001/authorization_requests.md`, AD-1; AD-3 below). | |
| 4 | + | |
| 5 | +## Statement (draft) | |
| 6 | + | |
| 7 | +> **Data.** The primary documents are Forms 10-K, 10-Q and 8-K retrieved from the U.S. Securities and Exchange Commission's EDGAR system (public domain; retrieved under the SEC fair-access policy, at most ten requests per second, with an identifying user agent) between May 2026 (S&P 500 issuers) and October 27, 2026 (remaining issuers); they are not redistributed because of their size (254.5 GB) and can be re-downloaded with `scripts/download_filings.py`. The issuer universe merges (i) the S&P 500 constituent table of Wikipedia scraped on May 25, 2026 (text under the Creative Commons Attribution-ShareAlike 4.0 licence) and (ii) the holdings file of the iShares Russell 3000 ETF (IWV) published by BlackRock, holdings as of October 23, 2026, downloaded on October 27, 2026 under the iShares website terms of use; the downloaded holdings file was not archived, so the 2,587 equity-line count of Table 1 is reported from the build log, while the mapped universe (`data/metadata/universe.json`, `russell3000_constituents.json`, `sp500_constituents.json`) is archived with SHA-256 checksums. The extraction outputs were generated with the OpenAI API (model alias `gpt-4o-mini` in June 2026, `gpt-4o-mini-2024-07-18` in October 2026) under OpenAI's Terms of Use; raw API responses were not stored, only the normalised mentions. The derived tables (`project_mentions`, `projects`, `project_timeline`, `extracted_sections`, `universe.json`) reside in DuckDB on the master node of the authors' cluster; **[author decision AD-1: released as Parquet with the paper / available from the authors on request]**. | |
| 8 | +> | |
| 9 | +> **Code.** The acquisition, parsing and extraction pipeline is revision `2324b58` of the private repository `uqo-research-db.git`; the files that define the extraction instrument (`spid/extract/llm.py`, `rules.py`, `taxonomy.py`, `resolve.py`) are archived with the paper (`code/audit/spid_snapshot/`, SHA-256 in `SHA256SUMS`). The exhibit code (`code/export_stats.py`, `code/make_tables.py`), the exports (`data/*.csv`, `data/stats.json`), the Stage D audit scripts (`code/audit/`) and their outputs (`results/`) are released with the paper; `run_all.sh` regenerates every table, figure and inline number from the exports and `tests/` checks that the exports reconcile. **[Licence — author decision AD-3: proposed MIT for code and CC BY 4.0 for the derived tables; no LICENSE file is committed until decided.]** | |
| 10 | +> | |
| 11 | +> **Reproducibility.** The LLM extraction is *re-runnable and ledgered*, not deterministic: the calls used temperature 0 without a seed, and the API's `system_fingerprint` was not recorded, so a re-run can return different mentions; what is reproducible from the archived materials is the mapping from the stored mentions to every table and number of this paper (verdict in `reproducibility/validation_report.md`). | |
| 12 | + | |
| 13 | +## Provenance table (DCAS rule 5) | |
| 14 | + | |
| 15 | +| source | URL / API | access date | licence / terms | restrictions | citation key | checksum | storage | size | | |
| 16 | +|---|---|---|---|---|---|---|---|---| | |
| 17 | +| SEC EDGAR primary documents (10-K, 10-Q, 8-K, 2010–2026-10-27) | `https://www.sec.gov/Archives/edgar/data/`, submissions index `https://data.sec.gov/submissions/CIK{cik}.json` | May 2026 (S&P 500), 2026-10-27 (full universe) | U.S. government work, public domain; fair-access policy | none; not redistributed (size) | `secfairaccess` | per-filing size recorded in `filings.file_size_bytes`; no file checksum stored (audit item) | M3U96b `~/UQO_RESEARCH_DB/data/raw/` | 254.5 GB | | |
| 18 | +| SEC `company_tickers.json` (ticker → CIK) | `https://www.sec.gov/files/company_tickers.json` | 2026-10-27 | public domain | snapshot **not archived** | — | — | — | — | | |
| 19 | +| iShares IWV holdings file `latest-holdings.csv` | `https://www.ishares.com/us/products/239714/ishares-russell-3000-etf` | 2026-10-27 (holdings as of 2026-10-23) | iShares website terms of use (personal, non-commercial use; no redistribution) | file **not archived**; mapped output archived | `iwvholdings2026` | `data/metadata/SHA256SUMS` (N10) for the derived JSON | M3U96b `data/metadata/russell3000_constituents.json` | 0.75 MB | | |
| 20 | +| Wikipedia, List of S&P 500 companies | `https://en.wikipedia.org/wiki/List_of_S%26P_500_companies` | 2026-05-25 | CC BY-SA 4.0 | none | `wikipediasp500` | `data/metadata/SHA256SUMS` (N10) | M3U96b `data/metadata/sp500_constituents.json` | 0.1 MB | | |
| 21 | +| FTSE Russell, Russell 3000 factsheet (2026-09-30) | see `docs/universe_coverage.md` | 2026-11-05 | FTSE Russell terms (read-only use) | figure reported, file not redistributed | `ftserussell3000factsheet2026` | `281e9264…0436e4` | `results/russell_coverage.json` | 0.26 MB | | |
| 22 | +| OpenAI API outputs (gpt-4o-mini) | `https://api.openai.com/v1/chat/completions` | 2026-06-17/18 and 2026-10-27/28 | OpenAI Terms of Use | raw JSON not stored | — | — | `project_mentions` (DuckDB) | 3.16 GB (spid.duckdb) | | |
| 23 | +| Corpus database `edgar_research.duckdb` | derived | 2026-10-28 | derived work (author decision) | large | — | mtime 2026-10-28 (N0 manifest) | M3U96b | 73.5 GB | | |
| 24 | +| Project database `spid.duckdb` | derived | 2026-10-28 12:21 | derived work (author decision AD-1) | — | — | N0 manifest | M3U96b | 3.16 GB | | |
| 25 | + | |
| 26 | +## Determinism paragraph (for §5.4 / §10.2; ECON-17, EDIT-11, AI_C-18 textual half) | |
| 27 | + | |
| 28 | +> The extraction calls used temperature 0, JSON-object response format and a fixed system prompt, but no `seed` parameter, and the `system_fingerprint` returned by the API was not stored; OpenAI documents that temperature 0 does not guarantee identical outputs across calls or model builds. The instrument is therefore *re-runnable and ledgered* — every section's submission is recorded with its mention count and timestamp — rather than deterministic, and the stability of the output under re-submission, prompt paraphrase and model substitution has not been measured (a 500-section protocol is specified in Section 8.2 and awaits authorisation). The proposed `extraction_runs` table records `seed`, `system_fingerprint`, prompt and taxonomy hashes per run. | |
| 29 | + | |
| 30 | +## Gate wording (Table 10; AI_C-14, ADVE-08) | |
| 31 | + | |
| 32 | +- gate (c) "unit tests": **Documented; not tested** — the pipeline snapshot contains no test file; the exhibit layer now has `tests/test_tables.py` (export reconciliation and regeneration), which does not test the pipeline. | |
| 33 | +- gate (f) "reproducible": **regenerable by the authors from the node; exports and code released** — until AD-1 decides the release of the derived tables. | |
modified
paper/main.bbl
+23 −15
@@ -3,12 +3,6 @@ | ||
| 3 | 3 | \fi |
| 4 | 4 | \begin{thebibliography}{xx} |
| 5 | 5 | |
| 6 | −\harvarditem[Bloom et al.]{Bloom, Hassan, Kalyani, Lerner and | |
| 7 | − Tahoun}{2021}{bloom2021} | |
| 8 | −{\bf Bloom, Nicholas, Tarek~A. Hassan, Aakash Kalyani, Josh Lerner, and Ahmed | |
| 9 | − Tahoun}, ``The Diffusion of Disruptive Technologies,'' NBER Working Paper | |
| 10 | − 28999, National Bureau of Economic Research 2021. | |
| 11 | − | |
| 12 | 6 | \harvarditem[Christen]{Christen}{2012}{christen2012} |
| 13 | 7 | {\bf Christen, Peter}, {\it Data Matching: Concepts and Techniques for Record |
| 14 | 8 | Linkage, Entity Resolution, and Duplicate Detection}, Berlin: Springer, 2012. |
@@ -29,9 +23,10 @@ | ||
| 29 | 23 | |
| 30 | 24 | \harvarditem[{FTSE Russell}]{{FTSE Russell}}{2026}{ftserussell2026} |
| 31 | 25 | {\bf {FTSE Russell}}, ``Russell {US} Equity Indexes: Construction and |
| 32 | − Methodology,'' \url{https://www.lseg.com/en/ftse-russell/indices/russell-us} | |
| 33 | − 2026. | |
| 34 | −\newblock Accessed October 2026. | |
| 26 | + Methodology, v7.2,'' | |
| 27 | + \url{https://www.lseg.com/content/dam/ftse-russell/en_us/documents/ground-rules/russell-us-indexes-construction-and-methodology.pdf} | |
| 28 | + August 2026. | |
| 29 | +\newblock Document dated August 2026; downloaded November 5, 2026. | |
| 35 | 30 | |
| 36 | 31 | \harvarditem[Gentzkow et al.]{Gentzkow, Kelly and Taddy}{2019}{gentzkow2019} |
| 37 | 32 | {\bf Gentzkow, Matthew, Bryan Kelly, and Matt Taddy}, ``Text as Data,'' {\it |
@@ -59,6 +54,14 @@ | ||
| 59 | 54 | Text-Based Analysis,'' {\it Review of Financial Studies}, 2015, {\it 28} (5), |
| 60 | 55 | 1312--1352. |
| 61 | 56 | |
| 57 | +\harvarditem[Kalyani et al.]{Kalyani, Bloom, Carvalho, Hassan, Lerner and | |
| 58 | + Tahoun}{2025}{kalyani2025} | |
| 59 | +{\bf Kalyani, Aakash, Nicholas Bloom, Marcela Carvalho, Tarek Hassan, Josh | |
| 60 | + Lerner, and Ahmed Tahoun}, ``The Diffusion of New Technologies,'' {\it | |
| 61 | + Quarterly Journal of Economics}, 2025, {\it 140} (2), 1299--1365. | |
| 62 | +\newblock Published version of NBER Working Paper 28999 (2021, DOI | |
| 63 | + 10.3386/w28999). | |
| 64 | + | |
| 62 | 65 | \harvarditem[Kelly et al.]{Kelly, Papanikolaou, Seru and |
| 63 | 66 | Taddy}{2021}{kelly2021} |
| 64 | 67 | {\bf Kelly, Bryan, Dimitris Papanikolaou, Amit Seru, and Matt Taddy}, |
@@ -96,14 +99,19 @@ | ||
| 96 | 99 | Kammoun}{2026}{spid2026} |
| 97 | 100 | {\bf Tandjambianda, Djerrycharli, Simon-Pierre Boucher, and Manel Kammoun}, |
| 98 | 101 | ``{SPID} --- {SEC} Project Intelligence Database: A Methodological Report on |
| 99 | − the {S\&P}~500 Release,'' Internal methodological report, Université du | |
| 100 | − Québec en Outaouais, Département des sciences administratives 2026. | |
| 101 | −\newblock Revision df7905d; October 28, 2026. | |
| 102 | + the {S\&P}~500 Release,'' Unpublished internal methodological report, | |
| 103 | + available from the authors, Université du Québec en Outaouais, Département | |
| 104 | + des sciences administratives 2026. | |
| 105 | +\newblock June 2026 study of revision \texttt{df7905d}; the report file was | |
| 106 | + regenerated on October 28, 2026. | |
| 102 | 107 | |
| 103 | 108 | \harvarditem[{U.S. Securities and Exchange Commission}]{{U.S. Securities and |
| 104 | − Exchange Commission}}{2024}{secfairaccess} | |
| 109 | + Exchange Commission}}{2026}{secfairaccess} | |
| 105 | 110 | {\bf {U.S. Securities and Exchange Commission}}, ``Accessing {EDGAR} Data: Fair |
| 106 | − Access Policy,'' \url{https://www.sec.gov/os/accessing-edgar-data} 2024. | |
| 107 | −\newblock Accessed October 2026. | |
| 111 | + Access Policy,'' | |
| 112 | + \url{https://www.sec.gov/search-filings/edgar-search-assistance/accessing-edgar-data} | |
| 113 | + 2026. | |
| 114 | +\newblock Accessed October 2026; URL re-verified November 5, 2026 (the former | |
| 115 | + \texttt{/os/accessing-edgar-data} address redirects here). | |
| 108 | 116 | |
| 109 | 117 | \end{thebibliography} |
modified
paper/main.pdf
+0 −0
Binary file not shown.
added
reproducibility/environment/requirements-exhibits.txt
+6 −0
@@ -0,0 +1,6 @@ | ||
| 1 | +# Exhibit layer (code/make_tables.py, tests/) — pins of the laptop environment that regenerated paper/tables and paper/figures on 2026-11-05. | |
| 2 | +# Python 3.14.4 (CPython, macOS 27.0 / Darwin 27.0.0, Apple Silicon). TeX: latexmk + pdflatex (TeX Live 2026), natbib + aer.bst. | |
| 3 | +# The node environment of code/export_stats.py and code/audit/ (M3U96b, uv-managed CPython 3.12.15, duckdb 2.0.0) is pinned by N12 in | |
| 4 | +# requirements.lock / .python-version / environment.md at the repository root. | |
| 5 | +matplotlib==3.10.9 | |
| 6 | +pytest==9.1.1 | |
added
reproducibility/validation_report.md
+34 −0
@@ -0,0 +1,34 @@ | ||
| 1 | +# Validation report — FIRC-0001, exhibit layer (protocol Part IX; Stage D laptop batches L1/L6) | |
| 2 | + | |
| 3 | +Prepared 2026-11-05 by the laptop executor on the review worktree `worktrees/FIRC-0001` (branch `review-20261103`). The node-side report (DuckDB audits, regeneration of the 29 exports by N13) is `UQO-WP-REVIEW/reproducibility/FIRC-0001/validation_report_N.md`; this report covers what was re-run on the laptop and the resulting verdict for the exhibit layer. | |
| 4 | + | |
| 5 | +## What was re-run, where, how long | |
| 6 | + | |
| 7 | +| step | command | machine | wall time | log | | |
| 8 | +|---|---|---|---|---| | |
| 9 | +| exhibits | `OMP_NUM_THREADS=4 nice -n 10 python3 -I code/make_tables.py` | laptop (Python 3.14.4, matplotlib 3.10.9) | ≈ 3 s | `reproducibility/FIRC-0001/logs/L1_make_tables_*.log`, `L1_build_*.log` | | |
| 10 | +| paper | `latexmk -pdf -interaction=nonstopmode main.tex` (pdflatex + bibtex, `aer.bst`) | laptop (TeX Live 2026) | ≈ 40 s | `L1_build_20261105-142858.log` | | |
| 11 | +| tests | `python3 -I -m pytest -q tests` | laptop (pytest 9.1.1) | ≈ 1 s | `L6_pytest_*.log` | | |
| 12 | +| DOI resolution | `curl` × 24 against Crossref / DataCite, 0.3 s apart | laptop | ≈ 20 s | `L4_crossref_20261105-141223.json` | | |
| 13 | + | |
| 14 | +No LLM or embedding call was made; nothing was written on any node; `data/stats.json` and the 29 committed CSV exports were not modified by any laptop step (the generator no longer rewrites `stats.json`; `sha256(data/stats.json)` is asserted unchanged by `tests/test_tables.py::test_regeneration_reproduces_committed_tables`). | |
| 15 | + | |
| 16 | +## What matched | |
| 17 | + | |
| 18 | +- **Export reconciliation (21 assertions, all pass):** the marginal CSV tables sum to the headline counts of `stats.json` — ledger rows 1,123,098; stored mentions 308,065 (by form, by year, by confidence); projects 121,099 (by type, status, span, mention bucket, first-seen year, sector, index membership); filings 527,548 (by form, by year × form, by year, by parse status); sections 1,342,283; universe 2,564 = 491 + 9 + 2,064 = 2,499 + 65 = Σ sectors; 10-K and 10-Q item tables sum to the per-form ledger rows and mentions; amount-bearing projects 56,500 in `amount_dist.csv`, `projects_by_type.csv` and `stats.json`. | |
| 19 | +- **Regeneration:** `make_tables.py --out <scratch>` reproduces every committed `paper/tables/*.tex` byte for byte and the committed `results/inline_numbers.json` (207 macros). Figures are regenerated as PDF (vector; `CreationDate` suppressed) — compared visually, not byte-wise. | |
| 20 | +- **Build:** `latexmk` exit 0, 0 LaTeX errors, 0 undefined citations/references, bibtex 0 warnings, 39 pages. Overfull boxes: 17 (baseline 14); the three new ones are the wider Tables 3 and 5 whose notes still sit inside the `adjustbox` (item LITE-10, Stage E/L3 moves the notes outside), and the 835 pt box is the hand-typed regex of `appendix.tex:19` that Stage E replaces by `\input{tables/gate_regex}`. | |
| 21 | +- **Prompt and gate regex:** `paper/tables/prompt_verbatim.tex` and `gate_regex.tex` are now generated from the frozen snapshot (`code/audit/spid_snapshot/llm.py`, `taxonomy.py`, `rules.py`, SHA-256 in `SHA256SUMS`) by parsing the source with `ast`; the printed regex concatenates to the `_PROJECT_GATE` pattern of `rules.py` exactly (three source lines, 26 alternatives), and the prompt lists the 21 taxonomy labels in the order `TYPE_LABELS` sends them. | |
| 22 | +- **Literals removed from the generators:** `make_tables.py` — `2{,}587` (now `data/documented_inputs.json:iwv_equity_lines_reported`, labelled "count reported by the build log, source file not archived"), `(500 CIKs)` (now `stats.json:universe.n_sp500` / `sp500_ciks`), `2010-01-04`/`2026-10-27` (now `stats.json:filing_date_range` or min/max of `filings_by_form.csv`), `128.5` (now `stats.json:amount_total_median_musd` or `amount_dist.csv`), `2700` (y-limit now `universe.n × 1.05`), `300` (label offset now 0.7 % of the maximum), `170` (now `documented_inputs.json:snippet_chars`); `export_stats.py` — the `spid_runs` strings (`2026-06 (gpt-4o-mini, 16 workers, 153,894 sections)`, `… 48 workers, 968,134 sections`) are now read by regex from the two run logs and from the ledger (`None` when absent). `tests/test_tables.py::test_make_tables_has_no_data_literals` tokenises the generator and fails on any integer literal of three or more digits other than the scale `100` (`1e3/1e6/1e9` are exponent forms). | |
| 23 | + | |
| 24 | +## What did not match / is still open | |
| 25 | + | |
| 26 | +- `export_stats.py` was **edited but not run** on the laptop (it needs the two DuckDB files on M3U96b); the new keys it exports (`filing_date_range`, `amount_total_median_musd`, `amount_max`, `amount_ge_50b`, `amount_le_zero`, `ledger_10k_mdna_median_words`, `annual_panel_firms`, `sp500_ciks`, `gate_pass_total`, `calls_by_form`, `backfilled_rows`, `outage`, `conf_exact_default_count`, `amount_raw_deviations`, `fiscal_year_null_mentions`, structured `spid_runs`) and its three new dumps (`ledger_collisions.csv`, `cohort_by_run.csv`, `calls_by_form.csv`, plus the seeded bootstrap `projects_by_type_ci.csv`) appear in the exhibits only once N13 has re-run it and its 29-file diff against the committed exports is empty. Until then `make_tables.py` falls back to the committed exports and to the N1/N3 result files (`results/ledger/*.json`, `results/resolver/*`), and the "95 % CI of the median" column of Table 5 prints `--`. | |
| 27 | +- The figure PNGs under `paper/figures/` are produced by no script (DATA-15); they are kept for the Word export only. | |
| 28 | +- The release of the derived DuckDB tables (AD-1) and the licence of code and tables (AD-3) are author decisions; no `LICENSE` file is committed. | |
| 29 | + | |
| 30 | +## Reproducibility verdict | |
| 31 | + | |
| 32 | +- **Exhibit layer (tables, figures, inline numbers from `data/` and `results/`): R1** — reproduced from the documented analysis data; the raw filings (254.5 GB) and the two DuckDB files are too large to ship and the LLM extraction is not re-run. | |
| 33 | +- **Export layer (`data/*.csv`, `stats.json` from the DuckDB files): provisional R2** until N13 reports its diff; R1 if the 29 committed files are regenerated byte-identically on M3U96b, R4 if any headline count moves. | |
| 34 | +- **Extraction layer (LLM calls): not reproducible by design** — re-runnable and ledgered, with the determinism statement of `docs/DATA_AVAILABILITY.md`; stability under re-submission is unmeasured (AR-LLM-1 not granted). | |
added
run_all.sh
+33 −0
@@ -0,0 +1,33 @@ | ||
| 1 | +#!/usr/bin/env bash | |
| 2 | +# Single entry point of the replication package (protocol Part IX). | |
| 3 | +# ./run_all.sh exhibits + tests + paper from the committed data/ exports (laptop, ~1 min) | |
| 4 | +# ./run_all.sh --with-node additionally regenerates data/ from the two DuckDB files (master node M3U96b, read-only) | |
| 5 | +# ./run_all.sh --with-audit additionally runs the Stage D read-only audits (code/audit/run_all.sh, node) | |
| 6 | +# Raw filings, the DuckDB files and the LLM extraction are never rebuilt here (see docs/DATA_AVAILABILITY.md). | |
| 7 | +set -euo pipefail | |
| 8 | +cd "$(dirname "$0")" | |
| 9 | +export OMP_NUM_THREADS=${OMP_NUM_THREADS:-4} | |
| 10 | +REPO=${REPO:-$HOME/UQO_RESEARCH_DB} | |
| 11 | +PY=${PY:-python3} | |
| 12 | + | |
| 13 | +for arg in "$@"; do | |
| 14 | + case "$arg" in | |
| 15 | + --with-node) | |
| 16 | + echo "== export_stats.py (read-only DuckDB, threads=4) -> data/" | |
| 17 | + nice -n 10 "$REPO/.venv/bin/python" -I code/export_stats.py "$REPO" data/ ;; | |
| 18 | + --with-audit) | |
| 19 | + echo "== Stage D audits -> results/" | |
| 20 | + nice -n 10 bash code/audit/run_all.sh ;; | |
| 21 | + *) echo "unknown option $arg" >&2; exit 2 ;; | |
| 22 | + esac | |
| 23 | +done | |
| 24 | + | |
| 25 | +echo "== make_tables.py -> paper/tables, paper/figures, paper/tables/numbers.tex, results/inline_numbers.json" | |
| 26 | +nice -n 10 "$PY" -I code/make_tables.py | |
| 27 | + | |
| 28 | +echo "== tests" | |
| 29 | +nice -n 10 "$PY" -I -m pytest -q tests | |
| 30 | + | |
| 31 | +echo "== paper" | |
| 32 | +( cd paper && nice -n 10 latexmk -pdf -interaction=nonstopmode main.tex > /dev/null && grep -c '^!' main.log || true ) | |
| 33 | +echo "done: paper/main.pdf" | |
added
tests/test_tables.py
+106 −0
@@ -0,0 +1,106 @@ | ||
| 1 | +"""Reproducibility checks for the exhibit layer (protocol Part IX). | |
| 2 | + | |
| 3 | +Run: python3 -I -m pytest -q tests | |
| 4 | +1. every CSV export reconciles with stats.json (sums of the marginal tables equal the headline counts); | |
| 5 | +2. code/make_tables.py regenerated into a scratch directory reproduces the committed paper/tables/*.tex byte for byte | |
| 6 | + and the committed results/inline_numbers.json; | |
| 7 | +3. make_tables.py never rewrites data/stats.json and contains no hand-typed data literal. | |
| 8 | +""" | |
| 9 | +import csv, hashlib, json, subprocess, sys, tokenize, io | |
| 10 | +from pathlib import Path | |
| 11 | +import pytest | |
| 12 | + | |
| 13 | +ROOT = Path(__file__).resolve().parent.parent | |
| 14 | +D = ROOT / "data" | |
| 15 | +S = json.load(open(D / "stats.json")) | |
| 16 | + | |
| 17 | + | |
| 18 | +def rows(name): | |
| 19 | + with open(D / f"{name}.csv", newline="") as f: | |
| 20 | + return list(csv.DictReader(f)) | |
| 21 | + | |
| 22 | + | |
| 23 | +def col_sum(name, col, where=None): | |
| 24 | + return sum(int(float(r[col])) for r in rows(name) if where is None or where(r)) | |
| 25 | + | |
| 26 | + | |
| 27 | +@pytest.mark.parametrize("name,col,key", [ | |
| 28 | + ("ledger_by_form", "sections", "ledger_total"), | |
| 29 | + ("mentions_by_form", "n", "mentions"), | |
| 30 | + ("mentions_by_year", "mentions", "mentions"), | |
| 31 | + ("confidence_dist", "n", "mentions"), | |
| 32 | + ("projects_by_type", "n", "projects"), | |
| 33 | + ("projects_by_status", "n", "projects"), | |
| 34 | + ("duration", "n", "projects"), | |
| 35 | + ("mentions_per_project", "n", "projects"), | |
| 36 | + ("projects_by_first_seen_year", "n", "projects"), | |
| 37 | + ("projects_by_sector", "n", "projects"), | |
| 38 | + ("projects_by_index", "projects", "projects"), | |
| 39 | + ("filings_by_form", "n", "filings_total"), | |
| 40 | + ("filings_by_year_form", "n", "filings_total"), | |
| 41 | + ("issuers_by_year", "filings", "filings_total"), | |
| 42 | + ("parse_status", "n", "filings_total"), | |
| 43 | + ("sections_by_form", "sections", "sections_total"), | |
| 44 | +]) | |
| 45 | +def test_csv_sums_reconcile_with_stats(name, col, key): | |
| 46 | + assert col_sum(name, col) == S[key], f"{name}.{col} does not sum to stats.json[{key}]" | |
| 47 | + | |
| 48 | + | |
| 49 | +def test_universe_reconciles(): | |
| 50 | + U = S["universe"] | |
| 51 | + assert U["both"] + U["sp_only"] + U["ru_only"] == U["n"] | |
| 52 | + assert U["n_russell"] == U["russell_distinct_ciks"] | |
| 53 | + assert S["issuers_with_filings"] + S["issuers_without_filings"] == U["n"] | |
| 54 | + assert col_sum("companies_by_sector", "n") == U["n"] | |
| 55 | + assert len(rows("issuers_without_filings")) == S["issuers_without_filings"] | |
| 56 | + | |
| 57 | + | |
| 58 | +def test_items_reconcile_with_ledger_by_form(): | |
| 59 | + lb = {r["form_type"]: r for r in rows("ledger_by_form")} | |
| 60 | + for form, name in (("10-K", "ledger_10k_items"), ("10-Q", "ledger_10q_items")): | |
| 61 | + assert col_sum(name, "n") == int(lb[form]["sections"]) | |
| 62 | + assert col_sum(name, "mentions") == int(lb[form]["mentions"]) | |
| 63 | + | |
| 64 | + | |
| 65 | +def test_amounts_and_fields(): | |
| 66 | + assert int(rows("amount_dist")[0]["n"]) == S["projects_fields"][0] | |
| 67 | + assert col_sum("projects_by_type", "with_amount") == S["projects_fields"][0] | |
| 68 | + assert S["mentions_fields"][-1] == S["mentions"] and S["projects_fields"][-1] == S["projects"] | |
| 69 | + | |
| 70 | + | |
| 71 | +def test_ledger_mentions_exceed_stored_mentions_by_collisions(): | |
| 72 | + # the ledger counts mentions before id de-duplication; stored mentions can only be fewer | |
| 73 | + assert col_sum("ledger_by_form", "mentions") >= S["mentions"] | |
| 74 | + assert col_sum("ledger_by_form", "with_mentions") <= S["ledger_total"] | |
| 75 | + | |
| 76 | + | |
| 77 | +def _sha(p): | |
| 78 | + return hashlib.sha256(p.read_bytes()).hexdigest() | |
| 79 | + | |
| 80 | + | |
| 81 | +def test_regeneration_reproduces_committed_tables(tmp_path): | |
| 82 | + before = _sha(D / "stats.json") | |
| 83 | + r = subprocess.run([sys.executable, "-I", str(ROOT / "code" / "make_tables.py"), "--out", str(tmp_path)], | |
| 84 | + capture_output=True, text=True, cwd=ROOT) | |
| 85 | + assert r.returncode == 0, r.stderr | |
| 86 | + assert _sha(D / "stats.json") == before, "make_tables.py must never rewrite stats.json" | |
| 87 | + mism = [] | |
| 88 | + for p in sorted((tmp_path / "tables").glob("*.tex")): | |
| 89 | + q = ROOT / "paper" / "tables" / p.name | |
| 90 | + if not q.exists() or q.read_text() != p.read_text(): | |
| 91 | + mism.append(p.name) | |
| 92 | + assert not mism, f"regenerated tables differ from the committed ones: {mism} (run python3 -I code/make_tables.py and commit)" | |
| 93 | + gen = json.load(open(tmp_path / "results" / "inline_numbers.json")) | |
| 94 | + com = json.load(open(ROOT / "results" / "inline_numbers.json")) | |
| 95 | + assert gen == com, "results/inline_numbers.json is stale" | |
| 96 | + | |
| 97 | + | |
| 98 | +def test_make_tables_has_no_data_literals(): | |
| 99 | + """Integer literals with three or more digits are not allowed in the generator (the scales 100, 1e3, 1e6, 1e9 are).""" | |
| 100 | + src = (ROOT / "code" / "make_tables.py").read_text() | |
| 101 | + bad = [] | |
| 102 | + for tok in tokenize.generate_tokens(io.StringIO(src).readline): | |
| 103 | + if tok.type == tokenize.NUMBER and "e" not in tok.string.lower() and "." not in tok.string and len(tok.string) >= 3 and tok.string != "100": | |
| 104 | + bad.append((tok.start[0], tok.string)) | |
| 105 | + assert not bad, f"numeric literals in make_tables.py: {bad}" | |
| 106 | + assert 'json.dump(S' not in src, "make_tables.py must not write stats.json" | |
| 107 | ||