Data and code availability statement — draft for §10.2 (L6, cluster L; FIRC0001-DATA-12, ADVE-08, AI_C-14, ECON-17)
Drafted 2026-11-05 (Stage D, laptop executor) following the AEA Data and Code Availability Standard (DCAS v1.0). Stage E inserts the statement in paper/sections/conclusion.tex §10.2; the two release decisions marked author decision are open (improvements/FIRC-0001/authorization_requests.md, AD-1; AD-3 below).
Statement (draft)
Data. The primary documents are Forms 10-K, 10-Q and 8-K retrieved from the U.S. Securities and Exchange Commission's EDGAR system (public domain; retrieved under the SEC fair-access policy, at most ten requests per second, with an identifying user agent) between May 2026 (S&P 500 issuers) and October 27, 2026 (remaining issuers); they are not redistributed because of their size (254.5 GB) and can be re-downloaded with
scripts/download_filings.py. The issuer universe merges (i) the S&P 500 constituent table of Wikipedia scraped on May 25, 2026 (text under the Creative Commons Attribution-ShareAlike 4.0 licence) and (ii) the holdings file of the iShares Russell 3000 ETF (IWV) published by BlackRock, holdings as of October 23, 2026, downloaded on October 27, 2026 under the iShares website terms of use; the downloaded holdings file was not archived, so the 2,587 equity-line count of Table 1 is reported from the build log, while the mapped universe (data/metadata/universe.json,russell3000_constituents.json,sp500_constituents.json) is archived with SHA-256 checksums. The extraction outputs were generated with the OpenAI API (model aliasgpt-4o-miniin June 2026,gpt-4o-mini-2024-07-18in October 2026) under OpenAI's Terms of Use; raw API responses were not stored, only the normalised mentions. The derived tables (project_mentions,projects,project_timeline,extracted_sections,universe.json) reside in DuckDB on the master node of the authors' cluster; [author decision AD-1: released as Parquet with the paper / available from the authors on request].Code. The acquisition, parsing and extraction pipeline is revision
2324b58of the private repositoryuqo-research-db.git; the files that define the extraction instrument (spid/extract/llm.py,rules.py,taxonomy.py,resolve.py) are archived with the paper (code/audit/spid_snapshot/, SHA-256 inSHA256SUMS). The exhibit code (code/export_stats.py,code/make_tables.py), the exports (data/*.csv,data/stats.json), the Stage D audit scripts (code/audit/) and their outputs (results/) are released with the paper;run_all.shregenerates every table, figure and inline number from the exports andtests/checks that the exports reconcile. [Licence — author decision AD-3: proposed MIT for code and CC BY 4.0 for the derived tables; no LICENSE file is committed until decided.]Reproducibility. The LLM extraction is re-runnable and ledgered, not deterministic: the calls used temperature 0 without a seed, and the API's
system_fingerprintwas not recorded, so a re-run can return different mentions; what is reproducible from the archived materials is the mapping from the stored mentions to every table and number of this paper (verdict inreproducibility/validation_report.md).
Provenance table (DCAS rule 5)
| source | URL / API | access date | licence / terms | restrictions | citation key | checksum | storage | size |
|---|---|---|---|---|---|---|---|---|
| SEC EDGAR primary documents (10-K, 10-Q, 8-K, 2010–2026-10-27) | https://www.sec.gov/Archives/edgar/data/, submissions index https://data.sec.gov/submissions/CIK{cik}.json |
May 2026 (S&P 500), 2026-10-27 (full universe) | U.S. government work, public domain; fair-access policy | none; not redistributed (size) | secfairaccess |
per-filing size recorded in filings.file_size_bytes; no file checksum stored (audit item) |
M3U96b ~/UQO_RESEARCH_DB/data/raw/ |
254.5 GB |
SEC company_tickers.json (ticker → CIK) |
https://www.sec.gov/files/company_tickers.json |
2026-10-27 | public domain | snapshot not archived | — | — | — | — |
iShares IWV holdings file latest-holdings.csv |
https://www.ishares.com/us/products/239714/ishares-russell-3000-etf |
2026-10-27 (holdings as of 2026-10-23) | iShares website terms of use (personal, non-commercial use; no redistribution) | file not archived; mapped output archived | iwvholdings2026 |
data/metadata/SHA256SUMS (N10) for the derived JSON |
M3U96b data/metadata/russell3000_constituents.json |
0.75 MB |
| Wikipedia, List of S&P 500 companies | https://en.wikipedia.org/wiki/List_of_S%26P_500_companies |
2026-05-25 | CC BY-SA 4.0 | none | wikipediasp500 |
data/metadata/SHA256SUMS (N10) |
M3U96b data/metadata/sp500_constituents.json |
0.1 MB |
| FTSE Russell, Russell 3000 factsheet (2026-09-30) | see docs/universe_coverage.md |
2026-11-05 | FTSE Russell terms (read-only use) | figure reported, file not redistributed | ftserussell3000factsheet2026 |
281e9264…0436e4 |
results/russell_coverage.json |
0.26 MB |
| OpenAI API outputs (gpt-4o-mini) | https://api.openai.com/v1/chat/completions |
2026-06-17/18 and 2026-10-27/28 | OpenAI Terms of Use | raw JSON not stored | — | — | project_mentions (DuckDB) |
3.16 GB (spid.duckdb) |
Corpus database edgar_research.duckdb |
derived | 2026-10-28 | derived work (author decision) | large | — | mtime 2026-10-28 (N0 manifest) | M3U96b | 73.5 GB |
Project database spid.duckdb |
derived | 2026-10-28 12:21 | derived work (author decision AD-1) | — | — | N0 manifest | M3U96b | 3.16 GB |
Determinism paragraph (for §5.4 / §10.2; ECON-17, EDIT-11, AI_C-18 textual half)
The extraction calls used temperature 0, JSON-object response format and a fixed system prompt, but no
seedparameter, and thesystem_fingerprintreturned by the API was not stored; OpenAI documents that temperature 0 does not guarantee identical outputs across calls or model builds. The instrument is therefore re-runnable and ledgered — every section's submission is recorded with its mention count and timestamp — rather than deterministic, and the stability of the output under re-submission, prompt paraphrase and model substitution has not been measured (a 500-section protocol is specified in Section 8.2 and awaits authorisation). The proposedextraction_runstable recordsseed,system_fingerprint, prompt and taxonomy hashes per run.
Gate wording (Table 10; AI_C-14, ADVE-08)
- gate (c) "unit tests": Documented; not tested — the pipeline snapshot contains no test file; the exhibit layer now has
tests/test_tables.py(export reconciliation and regeneration), which does not test the pipeline. - gate (f) "reproducible": regenerable by the authors from the node; exports and code released — until AD-1 decides the release of the derived tables.