SPB Git

spb/firc0001_uqo

Public

FIRC-0001 v1.0 (2026-10-28): Russell 3000 Strategic Project Database — baseline import before peer review

Simon-Pierre Boucher committed (Nov 5, 2026) Browse files

87 changed files +2,468 −0

87 changed files
added .gitignore +14 −0
@@ -0,0 +1,14 @@
1 +.DS_Store
2 +*.aux
3 +*.log
4 +*.fls
5 +*.fdb_latexmk
6 +*.synctex.gz
7 +*.out
8 +*.toc
9 +*.lof
10 +*.lot
11 +*.blg
12 +*.bcf
13 +*.run.xml
14 +__pycache__/
added README.md +20 −0
@@ -0,0 +1,20 @@
1 +# UQO Working Paper No. 0001 (FIRC-funded) — From Corporate Disclosures to a Russell 3000 Strategic Project Database
2 +
3 +*Identification, Measurement, Validation, and Research Applications* — Manel Kammoun, Djerrycharli Tandjambianda, Simon-Pierre Boucher (order to be confirmed; see `docs/contributions.md`). Version 1.0, October 28, 2026 — **working draft, research database under validation**. Research funded by the FIRC (Fonds institutionnel de recherche et de création, UQO).
4 +
5 +> **TL;DR** — SPID turns 527,548 SEC filings (10-K/10-Q/8-K, 2010–Oct 2026) of 2,564 issuers (S&P 500 ∪ iShares IWV Russell 3000 proxy) into 308,065 evidence-linked project mentions and 121,099 resolved candidate projects (21-category taxonomy, status, years, locations, technologies, partners, amounts). The paper documents the universe, the distributed acquisition/parsing architecture, the LLM extraction contract, the resolution rule and its failure modes, descriptive patterns, and a validation protocol — stating explicitly which gates are met and which are not.
6 +
7 +## Layout
8 +```
9 +paper/ main.tex, sections/, appendix/, tables/ (generated), figures/ (generated), references.bib → main.pdf
10 +code/ export_stats.py (run on the master node against the two DuckDB files) · make_tables.py (CSV → LaTeX/figures)
11 +data/ CSV exports + stats.json produced by export_stats.py (every number in the paper)
12 +docs/ implementation_audit.md · universe_definition.md · project_ontology.md · data_dictionary.md · contributions.md · research_log.md
13 +```
14 +Database and code: `uqo-research-db.git` (revision 2324b58) on the UQO Git server; raw filings are not redistributed.
15 +
16 +## Rebuild the exhibits
17 +```bash
18 +# on the master node: python code/export_stats.py ~/UQO_RESEARCH_DB data/
19 +python3 code/make_tables.py && cd paper && latexmk
20 +```
added Russell3000-Strategic-Project-Database_WP0001_v1.0_2026-10-28.docx +0 −0

Binary file not shown.

added code/export_stats.py +84 −0
@@ -0,0 +1,84 @@
1 +"""Export every number used in the working paper from the two DuckDB databases (run on the master node).
2 +
3 +Usage (on M3U96b): ~/UQO_RESEARCH_DB/.venv/bin/python export_stats.py ~/UQO_RESEARCH_DB <out_dir>
4 +Writes CSV files (one per table) + stats.json. Nothing is hand-edited downstream.
5 +"""
6 +import csv, json, sys
7 +from pathlib import Path
8 +import duckdb
9 +
10 +root = Path(sys.argv[1]); out = Path(sys.argv[2]); out.mkdir(parents=True, exist_ok=True)
11 +edgar = duckdb.connect(str(root / "data/duckdb/edgar_research.duckdb"), read_only=True)
12 +spid = duckdb.connect(str(root / "project-db/data/db/spid.duckdb"), read_only=True)
13 +# the src_* views stored in spid.duckdb point to the master node's parquet corpus (valid here)
14 +S = {}
15 +
16 +def dump(name, con, sql):
17 + cur = con.execute(sql); cols = [d[0] for d in cur.description]; rows = cur.fetchall()
18 + with open(out / f"{name}.csv", "w", newline="") as f:
19 + w = csv.writer(f); w.writerow(cols); w.writerows(rows)
20 + print(f"{name}: {len(rows)} rows"); return rows
21 +
22 +# ---- universe reconciliation -------------------------------------------------
23 +uni = json.load(open(root / "data/metadata/universe.json"))
24 +ru = json.load(open(root / "data/metadata/russell3000_constituents.json"))
25 +sp = json.load(open(root / "data/metadata/sp500_constituents.json"))
26 +S["universe"] = {"built_at": uni["built_at"], "n": uni["count"], "n_sp500": uni["n_sp500"], "n_russell": uni["n_russell3000"],
27 + "russell_holdings_as_of": ru["holdings_as_of"], "russell_fetched_at": ru["fetched_at"],
28 + "russell_equity_lines_mapped": ru["count"], "russell_unmatched": len(ru["unmatched"]),
29 + "russell_unmatched_list": ru["unmatched"], "sp500_fetched_at": sp["fetched_at"], "sp500_rows": sp["count"],
30 + "russell_distinct_ciks": len({c["cik"] for c in ru["constituents"]}),
31 + "both": sum(1 for c in uni["constituents"] if c["in_sp500"] and c["in_russell3000"]),
32 + "sp_only": sum(1 for c in uni["constituents"] if c["in_sp500"] and not c["in_russell3000"]),
33 + "ru_only": sum(1 for c in uni["constituents"] if not c["in_sp500"] and c["in_russell3000"])}
34 +S["issuers_with_filings"] = edgar.execute("select count(distinct cik) from filings").fetchone()[0]
35 +S["issuers_without_filings"] = edgar.execute("select count(*) from companies c where not exists (select 1 from filings f where f.cik=c.cik)").fetchone()[0]
36 +dump("issuers_without_filings", edgar, "select ticker, company_name, sector, in_sp500, in_russell3000 from companies c where not exists (select 1 from filings f where f.cik=c.cik) order by ticker")
37 +S["issuers_with_projects"] = spid.execute("select count(distinct cik) from projects").fetchone()[0]
38 +
39 +# ---- corpus -------------------------------------------------------------------
40 +dump("filings_by_form", edgar, "select form_type, count(*) n, min(filing_date) first, max(filing_date) last, round(sum(file_size_bytes)/1e9,1) gb from filings group by 1 order by 1")
41 +dump("filings_by_year_form", edgar, "select extract(year from filing_date)::int yr, form_type, count(*) n, count(distinct cik) issuers from filings group by 1,2 order by 1,2")
42 +dump("issuers_by_year", edgar, "select extract(year from filing_date)::int yr, count(distinct cik) issuers, count(*) filings from filings group by 1 order by 1")
43 +dump("parse_status", edgar, "select form_type, parse_status, count(*) n from filings group by 1,2 order by 1,2")
44 +dump("sections_by_form", edgar, "select f.form_type, count(*) sections, round(avg(s.word_count)) mean_words, round(median(s.word_count)) median_words from filing_sections s join filings f using(accession_number) group by 1 order by 1")
45 +dump("sections_10k_by_item", edgar, "select s.section_name, count(*) n, round(median(s.word_count)) median_words from filing_sections s join filings f using(accession_number) where f.form_type='10-K' group by 1 order by 2 desc")
46 +S["chunks"] = edgar.execute("select count(*), round(avg(word_count),1) from filing_chunks").fetchone()
47 +S["embeddings"] = edgar.execute("select count(*) from embeddings_metadata").fetchone()[0]
48 +S["sections_total"] = edgar.execute("select count(*) from filing_sections").fetchone()[0]
49 +S["filings_total"] = edgar.execute("select count(*) from filings").fetchone()[0]
50 +S["raw_gb"] = edgar.execute("select round(sum(file_size_bytes)/1e9,1) from filings").fetchone()[0]
51 +dump("companies_by_sector", edgar, "select sector, count(*) n, count(*) filter (where in_sp500) sp500, count(*) filter (where in_russell3000) russell from companies group by 1 order by 2 desc")
52 +dump("companies_by_shard", edgar, "select shard_node, count(*) n from companies group by 1 order by 1")
53 +dump("factor_tables", edgar, "select 'chunk_scores' t, count(*) n from chunk_scores union all select 'filing_scores', count(*) from filing_scores union all select 'annual_panel', count(*) from annual_panel union all select 'quarterly_panel', count(*) from quarterly_panel")
54 +
55 +# ---- extraction ledger --------------------------------------------------------
56 +dump("ledger_by_form", spid, "select v.form_type, count(*) sections, count(*) filter (where e.n_mentions>0) with_mentions, sum(e.n_mentions) mentions, count(distinct v.cik) issuers from extracted_sections e join v_sections v using(section_id) group by 1 order by 1")
57 +S["ledger_total"] = spid.execute("select count(*) from extracted_sections").fetchone()[0]
58 +dump("ledger_10k_items", spid, "select v.section_name, count(*) n, sum(e.n_mentions) mentions from extracted_sections e join v_sections v using(section_id) where v.form_type='10-K' group by 1 order by 2 desc")
59 +dump("ledger_10q_items", spid, "select v.section_name, count(*) n, sum(e.n_mentions) mentions from extracted_sections e join v_sections v using(section_id) where v.form_type='10-Q' group by 1 order by 2 desc")
60 +
61 +# ---- mentions / projects ------------------------------------------------------
62 +S["mentions"] = spid.execute("select count(*) from project_mentions").fetchone()[0]
63 +S["projects"] = spid.execute("select count(*) from projects").fetchone()[0]
64 +dump("mentions_by_form", spid, "select form_type, count(*) n, round(avg(confidence),3) mean_conf from project_mentions group by 1 order by 2 desc")
65 +dump("mentions_by_year", spid, "select extract(year from filing_date)::int yr, count(*) mentions, count(distinct cik) issuers from project_mentions group by 1 order by 1")
66 +dump("projects_by_type", spid, "select project_type, count(*) n, round(100.0*count(*)/(select count(*) from projects),1) pct, count(*) filter (where total_amount_usd is not null) with_amount, round(median(total_amount_usd)/1e6,1) median_amount_musd, round(avg(n_mentions),2) mean_mentions from projects group by 1 order by 2 desc")
67 +dump("projects_by_sector", spid, "select coalesce(sector,'(unknown)') sector, count(*) n, count(distinct cik) issuers, round(count(*)*1.0/count(distinct cik),1) per_issuer from projects group by 1 order by 2 desc")
68 +dump("projects_by_status", spid, "select status, count(*) n, round(100.0*count(*)/(select count(*) from projects),1) pct from projects group by 1 order by 2 desc")
69 +dump("projects_by_first_seen_year", spid, "select extract(year from first_seen)::int yr, count(*) n from projects group by 1 order by 1")
70 +dump("mentions_per_project", spid, "select case when n_mentions=1 then '1' when n_mentions=2 then '2' when n_mentions<=5 then '3-5' when n_mentions<=10 then '6-10' else '>10' end bucket, count(*) n from projects group by 1 order by min(n_mentions)")
71 +dump("projects_per_issuer", spid, "select quantile_cont(n, [0.1,0.25,0.5,0.75,0.9,0.99]) q, min(n) mn, max(n) mx, avg(n) mean from (select cik, count(*) n from projects group by 1)")
72 +dump("projects_by_index", spid, "select co.in_sp500, co.in_russell3000, count(*) projects, count(distinct p.cik) issuers from projects p join src_companies co on p.cik=co.cik group by 1,2")
73 +dump("confidence_dist", spid, "select round(confidence,1) conf, count(*) n from project_mentions group by 1 order by 1")
74 +dump("amount_dist", spid, "select quantile_cont(total_amount_usd, [0.1,0.25,0.5,0.75,0.9,0.99]) q, count(*) n, round(sum(total_amount_usd)/1e9,1) sum_gusd from projects where total_amount_usd is not null")
75 +dump("duration", spid, "select case when last_year is null or first_year is null then 'unknown' when last_year=first_year then '0' when last_year-first_year<=2 then '1-2' when last_year-first_year<=5 then '3-5' else '>5' end span_years, count(*) n from projects group by 1 order by 1")
76 +dump("kg", spid, "select 'nodes' k, node_type, count(*) n from kg_nodes group by 1,2 union all select 'edges', rel, count(*) from kg_edges group by 1,2 order by 1,3 desc")
77 +dump("top_issuers", spid, "select ticker, company_name, sector, count(*) projects from projects group by 1,2,3 order by 4 desc limit 15")
78 +dump("example_provenance", spid, "select p.project_id, p.ticker, p.project_name, p.project_type, p.status, p.total_amount_usd, p.n_mentions, p.first_seen, p.last_seen, t.accession_number, t.filing_date, t.form_type, t.status m_status, t.amount_usd, t.confidence, t.snippet from projects p join project_timeline t using(project_id) where p.n_mentions between 4 and 8 and p.total_amount_usd > 5e8 and p.project_type in ('plant_construction','data_center') order by p.n_filings desc, p.project_id, t.filing_date limit 24")
79 +S["mentions_fields"] = spid.execute("select count(*) filter (where amount_usd is not null) amt, count(*) filter (where len(locations)>0) loc, count(*) filter (where len(technologies)>0) tech, count(*) filter (where len(partners)>0) partner, count(*) filter (where len(years)>0) yrs, count(*) from project_mentions").fetchone()
80 +S["projects_fields"] = spid.execute("select count(*) filter (where total_amount_usd is not null), count(*) filter (where canonical_location is not null), count(*) filter (where len(technologies)>0), count(*) from projects").fetchone()
81 +S["spid_runs"] = {"sp500_run": "2026-06 (gpt-4o-mini, 16 workers, 153,894 sections)", "russell_run": "2026-10-27/28 (gpt-4o-mini-2024-07-18, 48 workers, 968,134 sections)"}
82 +S["extracted_at_range"] = [str(x) for x in spid.execute("select min(extracted_at), max(extracted_at) from extracted_sections").fetchone()]
83 +json.dump(S, open(out / "stats.json", "w"), indent=2, default=str)
84 +print(json.dumps(S, indent=1, default=str)[:3000])
added code/make_tables.py +195 −0
@@ -0,0 +1,195 @@
1 +"""Build every LaTeX table and figure of the paper from data/*.csv (no hand-typed numbers)."""
2 +import csv, json
3 +from pathlib import Path
4 +import matplotlib
5 +matplotlib.use("Agg")
6 +import matplotlib.pyplot as plt
7 +
8 +ROOT = Path(__file__).resolve().parent.parent
9 +D, T, F = ROOT / "data", ROOT / "paper" / "tables", ROOT / "paper" / "figures"
10 +T.mkdir(exist_ok=True); F.mkdir(exist_ok=True)
11 +S = json.load(open(D / "stats.json"))
12 +
13 +def rows(name):
14 + with open(D / f"{name}.csv") as f:
15 + return list(csv.DictReader(f))
16 +
17 +def n(x, d=0):
18 + x = float(x)
19 + return f"{x:,.{d}f}".replace(",", "{,}")
20 +
21 +def w(name, body):
22 + (T / name).write_text(body); print("table", name)
23 +
24 +LAB = {"ma_integration": "M\\&A / integration", "capex_program": "Capital expenditure program", "partnership": "Strategic partnership",
25 + "geographic_expansion": "Geographic expansion", "plant_construction": "Plant / site construction", "cost_reduction": "Cost reduction / restructuring",
26 + "drug_pipeline": "Drug / clinical pipeline", "product_launch": "New product / launch", "rd_program": "R\\&D program",
27 + "infrastructure": "Infrastructure investment", "manufacturing_expansion": "Manufacturing / capacity expansion",
28 + "technology_deployment": "Technology deployment", "digital_transformation": "Digital transformation", "erp_implementation": "ERP / systems implementation",
29 + "sustainability": "Sustainability / ESG", "supply_chain": "Supply-chain initiative", "data_center": "Data center", "energy_transition": "Energy transition / renewables",
30 + "ai_initiative": "AI / ML initiative", "automation": "Automation / robotics", "cloud_migration": "Cloud migration"}
31 +
32 +# ---- Table: universe reconciliation ----
33 +U = S["universe"]
34 +w("tab_reconciliation.tex", f"""\\begin{{tabular}}{{lr}}
35 +\\toprule
36 +Step & Count \\\\
37 +\\midrule
38 +iShares IWV holdings file, equity lines with a ticker (as of {U['russell_holdings_as_of']}) & 2{{,}}587 \\\\
39 +\\quad of which mapped to an SEC CIK (CVRs, escrow lines and FDIC-only banks excluded) & {n(U['russell_equity_lines_mapped'])} \\\\
40 +\\quad distinct CIKs (share classes collapsed) & {n(U['russell_distinct_ciks'])} \\\\
41 +S\\&P~500 constituents, cached snapshot of {U['sp500_fetched_at'][:10]} (500 CIKs) & {n(U['sp500_rows'])} \\\\
42 +\\midrule
43 +Merged issuer universe (distinct CIKs) & {n(U['n'])} \\\\
44 +\\quad in both lists & {n(U['both'])} \\\\
45 +\\quad S\\&P~500 only & {n(U['sp_only'])} \\\\
46 +\\quad Russell proxy only & {n(U['ru_only'])} \\\\
47 +\\midrule
48 +Issuers with at least one 10-K, 10-Q or 8-K since 2010 & {n(S['issuers_with_filings'])} \\\\
49 +Issuers without any such filing (foreign private issuers filing 20-F/40-F, FDIC-only banks) & {n(S['issuers_without_filings'])} \\\\
50 +Issuers with at least one resolved project & {n(S['issuers_with_projects'])} \\\\
51 +\\bottomrule
52 +\\end{{tabular}}""")
53 +
54 +# ---- Table: filings by form ----
55 +fb = rows("filings_by_form"); ps = rows("parse_status")
56 +err = {}
57 +for r in ps:
58 + err.setdefault(r["form_type"], {})[r["parse_status"]] = int(r["n"])
59 +body = "\\begin{tabular}{lrrrrr}\n\\toprule\nForm & Filings & First filing & Last filing & Raw HTML (GB) & Not parsed \\\\\n\\midrule\n"
60 +for r in fb:
61 + e = err.get(r["form_type"], {}); notp = e.get("error", 0) + e.get("missing", 0)
62 + body += f"{r['form_type']} & {n(r['n'])} & {r['first']} & {r['last']} & {float(r['gb']):.1f} & {notp} \\\\\n"
63 +body += f"\\midrule\nTotal & {n(S['filings_total'])} & 2010-01-04 & 2026-10-27 & {S['raw_gb']:.1f} & {sum(e.get('error',0)+e.get('missing',0) for e in err.values())} \\\\\n\\bottomrule\n\\end{{tabular}}"
64 +w("tab_filings.tex", body)
65 +
66 +# ---- Table: coverage by year ----
67 +iy = rows("issuers_by_year"); fy = rows("filings_by_year_form"); py = {r["yr"]: r for r in rows("projects_by_first_seen_year")}; my = {r["yr"]: r for r in rows("mentions_by_year")}
68 +byf = {}
69 +for r in fy: byf.setdefault(r["yr"], {})[r["form_type"]] = r["n"]
70 +body = "\\begin{tabular}{lrrrrrrr}\n\\toprule\nYear & Issuers & 10-K & 10-Q & 8-K & Mentions & Issuers w/ mentions & Projects first seen \\\\\n\\midrule\n"
71 +for r in iy:
72 + y = r["yr"]; b = byf[y]
73 + body += f"{y} & {n(r['issuers'])} & {n(b['10-K'])} & {n(b['10-Q'])} & {n(b['8-K'])} & {n(my[y]['mentions'])} & {n(my[y]['issuers'])} & {n(py[y]['n'])} \\\\\n"
74 +body += "\\bottomrule\n\\end{tabular}"
75 +w("tab_coverage_year.tex", body)
76 +
77 +# ---- Table: sections and extraction ledger ----
78 +sb = {r["form_type"]: r for r in rows("sections_by_form")}; lb = rows("ledger_by_form")
79 +body = "\\begin{tabular}{lrrrrrr}\n\\toprule\nForm & Parsed sections & Median words & Sections sent to the extractor & Sections with $\\geq 1$ mention & Mentions & Issuers \\\\\n\\midrule\n"
80 +tot = [0, 0, 0, 0]
81 +for r in lb:
82 + s = sb[r["form_type"]]
83 + body += f"{r['form_type']} & {n(s['sections'])} & {n(s['median_words'])} & {n(r['sections'])} & {n(r['with_mentions'])} & {n(r['mentions'])} & {n(r['issuers'])} \\\\\n"
84 + tot[0] += int(s["sections"]); tot[1] += int(r["sections"]); tot[2] += int(r["with_mentions"]); tot[3] += int(r["mentions"])
85 +body += f"\\midrule\nTotal & {n(tot[0])} & & {n(tot[1])} & {n(tot[2])} & {n(tot[3])} & {n(S['issuers_with_filings'])} \\\\\n\\bottomrule\n\\end{{tabular}}"
86 +w("tab_sections.tex", body)
87 +
88 +# ---- Table: 10-K / 10-Q items in scope ----
89 +body = "\\begin{tabular}{llrr}\n\\toprule\nForm & Section (item) & Sections extracted & Mentions \\\\\n\\midrule\n"
90 +for r in rows("ledger_10k_items"): body += f"10-K & \\texttt{{{r['section_name'].replace('_','\\_')}}} & {n(r['n'])} & {n(r['mentions'])} \\\\\n"
91 +body += "\\midrule\n"
92 +for r in rows("ledger_10q_items"): body += f"10-Q & \\texttt{{{r['section_name'].replace('_','\\_')}}} & {n(r['n'])} & {n(r['mentions'])} \\\\\n"
93 +body += "\\bottomrule\n\\end{tabular}"
94 +w("tab_items.tex", body)
95 +
96 +# ---- Table: projects by type ----
97 +pt = rows("projects_by_type")
98 +body = "\\begin{tabular}{llrrrrr}\n\\toprule\nCode & Category & Projects & Share (\\%) & With amount & Median amount (M USD) & Mentions / project \\\\\n\\midrule\n"
99 +for r in pt:
100 + body += f"\\texttt{{{r['project_type'].replace('_','\\_')}}} & {LAB[r['project_type']]} & {n(r['n'])} & {r['pct']} & {n(r['with_amount'])} & {n(r['median_amount_musd'],1)} & {float(r['mean_mentions']):.2f} \\\\\n"
101 +body += f"\\midrule\n & Total & {n(S['projects'])} & 100.0 & {n(S['projects_fields'][0])} & 128.5 & {S['mentions']/S['projects']:.2f} \\\\\n\\bottomrule\n\\end{{tabular}}"
102 +w("tab_types.tex", body)
103 +
104 +# ---- Table: by sector ----
105 +cs = {r["sector"]: r for r in rows("companies_by_sector")}
106 +body = "\\begin{tabular}{lrrrrr}\n\\toprule\nGICS sector (iShares) & Issuers in universe & of which S\\&P~500 & Issuers with projects & Projects & Projects per issuer \\\\\n\\midrule\n"
107 +for r in rows("projects_by_sector"):
108 + c = cs.get(r["sector"], {"n": "", "sp500": ""})
109 + body += f"{r['sector']} & {n(c['n']) if c['n'] else ''} & {n(c['sp500']) if c['sp500'] else ''} & {n(r['issuers'])} & {n(r['n'])} & {float(r['per_issuer']):.1f} \\\\\n"
110 +body += "\\bottomrule\n\\end{tabular}"
111 +w("tab_sectors.tex", body)
112 +
113 +# ---- Table: status, mention buckets, duration, confidence ----
114 +st = rows("projects_by_status"); mb = rows("mentions_per_project"); du = rows("duration"); cf = rows("confidence_dist")
115 +body = "\\begin{tabular}{lr@{\\hspace{2em}}lr@{\\hspace{2em}}lr}\n\\toprule\nLatest status & Projects & Mentions per project & Projects & Span of cited years & Projects \\\\\n\\midrule\n"
116 +du_order = ["0", "1-2", "3-5", ">5", "unknown"]; dus = {r["span_years"]: r["n"] for r in du}
117 +rowsn = max(len(st), len(mb), len(du_order))
118 +for i in range(rowsn):
119 + a = f"\\texttt{{{st[i]['status'].replace('_','\\_')}}} & {n(st[i]['n'])}" if i < len(st) else "&"
120 + b = f"{mb[i]['bucket']} & {n(mb[i]['n'])}" if i < len(mb) else "&"
121 + c = f"{du_order[i].replace('>','$>$')} years & {n(dus[du_order[i]])}" if i < len(du_order) else "&"
122 + body += f"{a} & {b} & {c} \\\\\n"
123 +body += "\\bottomrule\n\\end{tabular}"
124 +w("tab_status.tex", body)
125 +body = "\\begin{tabular}{lr}\n\\toprule\nSelf-reported confidence (rounded) & Mentions \\\\\n\\midrule\n"
126 +for r in cf: body += f"{r['conf']} & {n(r['n'])} \\\\\n"
127 +body += "\\bottomrule\n\\end{tabular}"
128 +w("tab_confidence.tex", body)
129 +
130 +# ---- Table: amounts ----
131 +ad = rows("amount_dist")[0]; q = json.loads(ad["q"]); ppi = json.loads(rows("projects_per_issuer")[0]["q"])
132 +body = f"""\\begin{{tabular}}{{lrrrrrr}}
133 +\\toprule
134 + & P10 & P25 & Median & P75 & P90 & P99 \\\\
135 +\\midrule
136 +Largest amount cited per project, M USD ($N$ = {n(ad['n'])}) & {q[0]/1e6:,.1f} & {q[1]/1e6:,.0f} & {q[2]/1e6:,.1f} & {q[3]/1e6:,.0f} & {q[4]/1e6:,.0f} & {q[5]/1e6:,.0f} \\\\
137 +Projects per issuer ($N$ = {n(S['issuers_with_projects'])}) & {ppi[0]:.0f} & {ppi[1]:.0f} & {ppi[2]:.0f} & {ppi[3]:.0f} & {ppi[4]:.0f} & {ppi[5]:.0f} \\\\
138 +\\bottomrule
139 +\\end{{tabular}}""".replace(",", "{,}")
140 +w("tab_amounts.tex", body)
141 +
142 +# ---- Table: knowledge graph ----
143 +kg = rows("kg")
144 +body = "\\begin{tabular}{llr}\n\\toprule\n & Type & Count \\\\\n\\midrule\n"
145 +for r in kg: body += f"{r['k'].capitalize()} & \\texttt{{{r['node_type'].replace('_','\\_')}}} & {n(r['n'])} \\\\\n"
146 +body += "\\bottomrule\n\\end{tabular}"
147 +w("tab_graph.tex", body)
148 +
149 +# ---- Table: top issuers ----
150 +body = "\\begin{tabular}{llr}\n\\toprule\nTicker & Issuer & Resolved projects \\\\\n\\midrule\n"
151 +for r in rows("top_issuers")[:12]: body += f"{r['ticker']} & {r['company_name'].title().replace('&','\\&')} & {n(r['projects'])} \\\\\n"
152 +body += "\\bottomrule\n\\end{tabular}"
153 +w("tab_top_issuers.tex", body)
154 +
155 +# ---- Table: provenance example (Cheniere Corpus Christi) ----
156 +ex = [r for r in rows("example_provenance") if r["ticker"] == "LNG"]
157 +body = "\\begin{tabular}{llllp{6.2cm}}\n\\toprule\nAccession & Filed & Form & Status / amount & Evidence snippet (model-generated description grounded in the section) \\\\\n\\midrule\n"
158 +for r in ex[:8]:
159 + amt = f"{float(r['amount_usd'])/1e9:.1f} B USD" if r["amount_usd"] else "--"
160 + snip = r["snippet"].replace("&", "\\&").replace("%", "\\%").replace("$", "\\$")[:170]
161 + body += f"\\texttt{{{r['accession_number']}}} & {r['filing_date']} & {r['form_type']} & {r['m_status'].replace('_','\\_')} / {amt} & {snip} \\\\\n"
162 +body += "\\bottomrule\n\\end{tabular}"
163 +w("tab_provenance.tex", body)
164 +S["example_project"] = {k: ex[0][k] for k in ("project_id", "ticker", "project_name", "project_type", "status", "total_amount_usd", "n_mentions", "first_seen", "last_seen")} if ex else {}
165 +
166 +# ---- Figures ----
167 +BLUE, GOLD, GREY = "#16365c", "#c9a227", "#8a8a8a"
168 +yrs = [int(r["yr"]) for r in iy]
169 +fig, ax1 = plt.subplots(figsize=(7.2, 3.4))
170 +ax1.bar(yrs, [int(r["filings"]) for r in iy], color=BLUE, alpha=.85, label="Filings (10-K, 10-Q, 8-K)")
171 +ax1.set_ylabel("Filings per year"); ax1.set_xlabel("Filing year")
172 +ax2 = ax1.twinx(); ax2.plot(yrs, [int(r["issuers"]) for r in iy], color=GOLD, marker="o", lw=2, label="Issuers with $\\geq$1 filing")
173 +ax2.set_ylabel("Issuers"); ax2.set_ylim(0, 2700)
174 +h1, l1 = ax1.get_legend_handles_labels(); h2, l2 = ax2.get_legend_handles_labels(); ax1.legend(h1 + h2, l1 + l2, loc="upper left", frameon=False, fontsize=8)
175 +for a in (ax1, ax2): a.spines[["top"]].set_visible(False)
176 +fig.tight_layout(); fig.savefig(F / "fig_coverage.pdf"); plt.close(fig)
177 +
178 +fig, ax = plt.subplots(figsize=(7.2, 5.2))
179 +labs = [LAB[r["project_type"]].replace("\\&", "&") for r in pt][::-1]; vals = [int(r["n"]) for r in pt][::-1]
180 +ax.barh(labs, vals, color=BLUE); ax.set_xlabel("Resolved projects"); ax.spines[["top", "right"]].set_visible(False)
181 +for i, v in enumerate(vals): ax.text(v + 300, i, f"{v:,}", va="center", fontsize=7)
182 +fig.tight_layout(); fig.savefig(F / "fig_types.pdf"); plt.close(fig)
183 +
184 +fig, ax = plt.subplots(figsize=(7.2, 3.2))
185 +ax.plot(yrs, [int(my[str(y)]["mentions"]) for y in yrs], color=BLUE, marker="o", lw=2, label="Project mentions")
186 +ax.plot(yrs, [int(py[str(y)]["n"]) for y in yrs], color=GOLD, marker="s", lw=2, label="Projects first observed")
187 +ax.set_xlabel("Filing year"); ax.set_ylabel("Count"); ax.legend(frameon=False, fontsize=8); ax.spines[["top", "right"]].set_visible(False)
188 +fig.tight_layout(); fig.savefig(F / "fig_mentions_year.pdf"); plt.close(fig)
189 +
190 +fig, ax = plt.subplots(figsize=(7.2, 3.0))
191 +ax.bar([r["conf"] for r in cf], [int(r["n"]) for r in cf], width=0.08, color=BLUE)
192 +ax.set_xlabel("Self-reported extraction confidence"); ax.set_ylabel("Mentions"); ax.spines[["top", "right"]].set_visible(False)
193 +fig.tight_layout(); fig.savefig(F / "fig_confidence.pdf"); plt.close(fig)
194 +print("figures ok")
195 +json.dump(S, open(D / "stats.json", "w"), indent=2, default=str)
added code/tex2docx/post_docx.py +94 −0
@@ -0,0 +1,94 @@
1 +import sys, re, datetime
2 +from docx import Document
3 +from docx.shared import Pt
4 +from docx.enum.text import WD_ALIGN_PARAGRAPH
5 +path = sys.argv[1]; doc = Document(path)
6 +paras = doc.paragraphs
7 +norm = lambda p: re.sub(r"\s+", " ", p.text.replace(" ", " ")).strip()
8 +title = "From Corporate Disclosures to a Russell 3000 Strategic Project Database"
9 +# --- title page: everything before the Abstract block is centred
10 +i_abs = next(i for i, p in enumerate(paras) if norm(p) == "Abstract")
11 +for p in paras[:i_abs - 1]:
12 + p.alignment = WD_ALIGN_PARAGRAPH.CENTER
13 + if norm(p) == title:
14 + for r in p.runs: r.font.size = Pt(18); r.font.bold = True
15 + elif norm(p).startswith("Identification, Measurement"):
16 + for r in p.runs: r.font.size = Pt(13)
17 +# --- move References (pandoc appends it at the end) before the appendix, as in the PDF
18 +h1 = [p for p in paras if p.style.name == "Heading 1"]
19 +refs = next(p for p in h1 if norm(p) == "References")
20 +appA = next(p for p in h1 if re.match(r"^A [A-Z]", norm(p)))
21 +body = doc.element.body; kids = list(body)
22 +i_refs, i_app = kids.index(refs._p), kids.index(appA._p)
23 +moved = [k for k in kids[i_refs:] if k.tag.endswith("}sectPr") is False]
24 +for k in moved: appA._p.addprevious(k)
25 +# --- page breaks where the LaTeX source has \newpage
26 +breaks = 0
27 +paras[i_abs - 1].paragraph_format.page_break_before = True; breaks += 1
28 +for p in h1:
29 + t = norm(p)
30 + if t.startswith("1 Introduction") or t == "References" or p is appA:
31 + p.paragraph_format.page_break_before = True; breaks += 1
32 +# --- document properties
33 +cp = doc.core_properties
34 +cp.title = title
35 +cp.subject = "Identification, Measurement, Validation, and Research Applications"
36 +cp.author = "Manel Kammoun; Djerrycharli Tandjambianda; Simon-Pierre Boucher"
37 +cp.keywords = "SEC filings, text as data, corporate investment, strategic projects, information extraction, large language models, entity resolution, Russell 3000"
38 +cp.comments = "UQO Working Paper No. 0001 (FIRC-funded), Version 1.0, October 28, 2026 (working draft under validation)"
39 +cp.created = cp.modified = datetime.datetime(2026, 10, 28, 12, 0)
40 +# --- base fonts closer to the PDF (Times)
41 +from docx.oxml.ns import qn
42 +for st in doc.styles:
43 + try:
44 + if st.type == 1 and st.font is not None and (st.name == "Normal" or st.name.startswith("Heading") or st.name in ("Title", "Body Text", "First Paragraph", "Compact", "Caption", "Bibliography", "Block Text")):
45 + st.font.name = "Times New Roman"
46 + rpr = st.element.get_or_add_rPr(); rf = rpr.find(qn("w:rFonts"))
47 + if rf is None:
48 + rf = rpr.makeelement(qn("w:rFonts"), {}); rpr.append(rf)
49 + for a in ("w:ascii", "w:hAnsi", "w:eastAsia", "w:cs"): rf.set(qn(a), "Times New Roman")
50 + if st.name in ("Normal", "Body Text", "First Paragraph", "Compact"): st.font.size = Pt(12)
51 + except Exception:
52 + pass
53 +# --- wide tables: smaller font and tight cell padding so that 8-10 column tables fit the page width
54 +from docx.shared import Inches
55 +from docx.oxml import OxmlElement
56 +for sec in doc.sections:
57 + sec.left_margin = sec.right_margin = Inches(0.85)
58 +for tbl in doc.tables:
59 + ncol = max(len(r.cells) for r in tbl.rows)
60 + if ncol >= 7:
61 + size = Pt(7) if ncol >= 9 else Pt(8)
62 + tbl.autofit = True
63 + for row in tbl.rows:
64 + for cell in row.cells:
65 + for para in cell.paragraphs:
66 + para.paragraph_format.space_after = Pt(0); para.paragraph_format.space_before = Pt(0)
67 + for run in para.runs: run.font.size = size
68 + tcPr = cell._tc.get_or_add_tcPr(); mar = OxmlElement("w:tcMar")
69 + for side in ("left", "right"):
70 + el = OxmlElement(f"w:{side}"); el.set(qn("w:w"), "30"); el.set(qn("w:type"), "dxa"); mar.append(el)
71 + tcPr.append(mar)
72 +# --- proportional column widths for wide tables (pandoc emits equal 0.55-inch grid columns)
73 +TOTAL = 9792 # twips: 8.5in - 2 x 0.85in margins
74 +for tbl in doc.tables:
75 + ncol = max(len(r.cells) for r in tbl.rows)
76 + if ncol < 7: continue
77 + lens = [1] * ncol
78 + for row in tbl.rows:
79 + for j, cell in enumerate(row.cells[:ncol]):
80 + lens[j] = max(lens[j], min(28, max((len(w) for w in cell.text.split()), default=1)))
81 + weights = [max(l, 4) for l in lens]; tot = sum(weights); widths = [int(TOTAL * wgt / tot) for wgt in weights]
82 + grid = tbl._tbl.find(qn("w:tblGrid"))
83 + if grid is not None:
84 + for g, wd in zip(grid.findall(qn("w:gridCol")), widths): g.set(qn("w:w"), str(wd))
85 + tblPr = tbl._tbl.tblPr
86 + for e in tblPr.findall(qn("w:tblW")): e.set(qn("w:type"), "dxa"); e.set(qn("w:w"), str(TOTAL))
87 + for e in tblPr.findall(qn("w:tblLayout")): e.set(qn("w:type"), "fixed")
88 + for row in tbl.rows:
89 + for j, cell in enumerate(row.cells[:ncol]):
90 + tcPr = cell._tc.get_or_add_tcPr(); tcW = tcPr.find(qn("w:tcW"))
91 + if tcW is None: tcW = OxmlElement("w:tcW"); tcPr.insert(0, tcW)
92 + tcW.set(qn("w:type"), "dxa"); tcW.set(qn("w:w"), str(widths[j]))
93 +doc.save(path)
94 +print(f"post-processed: {i_abs-1} title-page paragraphs centred, {len(moved)} reference paragraphs moved before appendix, {breaks} page breaks, properties set")
added code/tex2docx/prep_tex.py +168 −0
@@ -0,0 +1,168 @@
1 +"""Flatten a LaTeX working paper and resolve numbering from main.aux so that
2 +pandoc produces a Word file with correct section/table/figure/equation numbers."""
3 +import re, sys, pathlib
4 +
5 +PAPER = pathlib.Path(sys.argv[1]).resolve()
6 +OUT = pathlib.Path(sys.argv[2])
7 +warn = lambda *a: print("WARN:", *a, file=sys.stderr)
8 +
9 +def strip_comments(txt):
10 + out = []
11 + for line in txt.splitlines():
12 + if re.match(r'^\s*%', line):
13 + continue
14 + # drop trailing unescaped % comments
15 + line = re.sub(r'(?<!\\)%.*$', '', line)
16 + out.append(line)
17 + return "\n".join(out) + "\n"
18 +
19 +def inline(path):
20 + txt = strip_comments(path.read_text())
21 + def rep(m):
22 + name = m.group(1)
23 + p = PAPER / (name if name.endswith('.tex') else name + '.tex')
24 + if not p.exists():
25 + warn("missing input", p); return m.group(0)
26 + return inline(p)
27 + return re.sub(r'\\input\{([^}]+)\}', rep, txt)
28 +
29 +def brace_arg(s, i):
30 + """s[i] == '{' -> return (content, index after closing brace)"""
31 + assert s[i] == '{', s[i:i+20]
32 + depth = 0
33 + for j in range(i, len(s)):
34 + if s[j] == '{' and (j == 0 or s[j-1] != '\\'): depth += 1
35 + elif s[j] == '}' and s[j-1] != '\\':
36 + depth -= 1
37 + if depth == 0:
38 + return s[i+1:j], j+1
39 + raise ValueError("unbalanced braces")
40 +
41 +# ---------------------------------------------------------------- aux
42 +aux = (PAPER / 'main.aux').read_text()
43 +labels = {}
44 +for m in re.finditer(r'\\newlabel\{([^}]+)\}\{', aux):
45 + outer, _ = brace_arg(aux, m.end()-1)
46 + num, _ = brace_arg(outer, 0)
47 + labels[m.group(1)] = num.strip()
48 +toc = []
49 +for m in re.finditer(r'\\contentsline \{(section|subsection|subsubsection)\}\{\\numberline \{([^}]*)\}(.*?)\}\{\d+\}\{', aux):
50 + toc.append((m.group(1), m.group(2), m.group(3)))
51 +print(f"aux: {len(labels)} labels, {len(toc)} numbered headings")
52 +
53 +# ---------------------------------------------------------------- flatten
54 +tex = inline(PAPER / 'main.tex')
55 +
56 +# ---- WP 28 shims ----------------------------------------------------------
57 +# TikZ pipeline figure -> pre-rendered PNG
58 +tex = re.sub(r'\\begin\{tikzpicture\}.*?\\end\{tikzpicture\}', r'\\includegraphics[width=\\textwidth]{figures/fig_pipeline.png}', tex, flags=re.S)
59 +# PDF figures -> PNG (pandoc/docx cannot embed PDF)
60 +tex = re.sub(r'\\includegraphics(\[[^\]]*\])?\{figures/([^}]+)\.pdf\}', r'\\includegraphics\1{figures/\2.png}', tex)
61 +# \adjustbox{max width=...}{...} wrapper -> content only
62 +def strip_adjustbox(t):
63 + out = []; pos = 0
64 + while True:
65 + m = re.search(r'\\adjustbox\{[^{}]*\}\{', t[pos:])
66 + if not m: break
67 + start = pos + m.start(); open_ = pos + m.end() - 1
68 + content, end = brace_arg(t, open_)
69 + out.append(t[pos:start]); out.append(content); pos = end
70 + out.append(t[pos:]); return "".join(out)
71 +tex = strip_adjustbox(tex)
72 +# threeparttable / tablenotes -> plain notes paragraph
73 +tex = re.sub(r'\\(begin|end)\{threeparttable\}', '', tex)
74 +tex = re.sub(r'\\begin\{tablenotes\}\\footnotesize\s*\\item\s*(.*?)\\end\{tablenotes\}', lambda m: '\\par\\textit{Notes:} ' + m.group(1).strip(), tex, flags=re.S)
75 +# prompt block (quote + obeylines) -> verbatim
76 +tex = re.sub(r'\\begin\{quote\}\\footnotesize\\ttfamily.*?\\end\{quote\}', lambda m: (PAPER / 'tables' / 'prompt_verbatim.tex').read_text(), tex, flags=re.S)
77 +tex = re.sub(r'\\input\{tables/prompt\}', lambda m: (PAPER / 'tables' / 'prompt_verbatim.tex').read_text(), tex)
78 +# tabularx -> tabular (pandoc handles tabular more reliably); X -> l
79 +def fix_tabularx(m):
80 + spec = m.group(1).replace('X', 'l')
81 + return '\\begin{tabular}{' + spec + '}'
82 +tex = re.sub(r'\\begin\{tabularx\}\{\\textwidth\}\{([^}]*)\}', fix_tabularx, tex)
83 +tex = tex.replace('\\end{tabularx}', '\\end{tabular}')
84 +tex = tex.replace('\\allowbreak', '')
85 +tex = re.sub(r'\\setlength\{\\emergencystretch\}\{[^}]*\}', '', tex)
86 +tex = re.sub(r'\\begin\{tabular\}\{lr@\{\\hspace\{2em\}\}lr@\{\\hspace\{2em\}\}lr\}', r'\\begin{tabular}{lrlrlr}', tex)
87 +tex = re.sub(r'\\begin\{tabular\}\{llllp\{[^}]*\}\}', r'\\begin{tabular}{lllll}', tex)
88 +
89 +# titlepage: pandoc cannot evaluate \ifx -> keep the \else branch only
90 +tex = re.sub(r'\\ifx\\WPsubtitle\\empty\\else(.*?)\\fi', r'\1', tex, flags=re.S)
91 +
92 +# appendix environment / counters that pandoc does not understand
93 +tex = re.sub(r'\\(begin|end)\{appendices\}', '', tex)
94 +tex = re.sub(r'\\renewcommand\{\\the(table|figure)\}\{[^{}]*(\{[^{}]*\})*[^{}]*\}', '', tex)
95 +tex = re.sub(r'\\setcounter\{(table|figure)\}\{0\}', '', tex)
96 +
97 +# ---------------------------------------------------------------- headings
98 +pos = 0; out = []; ti = 0
99 +pat = re.compile(r'\\(section|subsection|subsubsection)\{')
100 +while True:
101 + m = pat.search(tex, pos)
102 + if not m: break
103 + title, end = brace_arg(tex, m.end()-1)
104 + out.append(tex[pos:m.start()])
105 + lvl = m.group(1)
106 + if ti < len(toc) and toc[ti][0] == lvl:
107 + num = toc[ti][1]; ti += 1
108 + if re.sub(r'\W', '', toc[ti-1][2])[:12] != re.sub(r'\W', '', title)[:12]:
109 + warn("heading mismatch:", repr(toc[ti-1][2][:40]), "vs", repr(title[:40]))
110 + out.append(f"\\{lvl}{{{num}~{title}}}")
111 + else:
112 + warn("unnumbered heading kept:", lvl, title[:40])
113 + out.append(tex[m.start():end])
114 + pos = end
115 +out.append(tex[pos:]); tex = "".join(out)
116 +if ti != len(toc): warn(f"only {ti}/{len(toc)} toc entries consumed")
117 +
118 +# ---------------------------------------------------------------- floats: caption prefix
119 +def fix_float(m):
120 + kind, body = m.group(1), m.group(2)
121 + labs = [l for l in re.findall(r'\\label\{([^}]+)\}', body) if l in labels]
122 + if not labs:
123 + warn("float without known label", body[:60]); return m.group(0)
124 + num = labels[labs[0]]
125 + word = 'Table' if kind == 'table' else 'Figure'
126 + cm = re.search(r'\\caption(\[[^\]]*\])?\{', body)
127 + if not cm: return m.group(0)
128 + cap, end = brace_arg(body, cm.end()-1)
129 + body = body[:cm.start()] + f"\\caption{{{word} {num}: {cap}}}" + body[end:]
130 + return f"\\begin{{{kind}}}{body}\\end{{{kind}}}"
131 +tex = re.sub(r'\\begin\{(table|figure)\}(.*?)\\end\{\1\}', fix_float, tex, flags=re.S)
132 +
133 +# ---------------------------------------------------------------- equations: number tag
134 +def fix_eq(m):
135 + body = m.group(1)
136 + lm = re.search(r'\\label\{([^}]+)\}', body)
137 + if lm and lm.group(1) in labels:
138 + body = body.replace(lm.group(0), '')
139 + body = body.rstrip() + f"\\qquad\\text{{({labels[lm.group(1)]})}}\n"
140 + return "\\begin{equation}" + body + "\\end{equation}"
141 +tex = re.sub(r'\\begin\{equation\}(.*?)\\end\{equation\}', fix_eq, tex, flags=re.S)
142 +
143 +# ---------------------------------------------------------------- cross-references
144 +missing = set()
145 +def ref(m):
146 + k = m.group(2)
147 + if k not in labels:
148 + missing.add(k); return '??'
149 + return f"({labels[k]})" if m.group(1) == 'eqref' else labels[k]
150 +tex = re.sub(r'\\(eqref|ref)\{([^}]+)\}', ref, tex)
151 +if missing: warn("unresolved labels:", sorted(missing))
152 +tex = re.sub(r'\\label\{[^}]+\}', '', tex)
153 +
154 +# booktabs partial rules have no Word equivalent and leak as text
155 +tex = re.sub(r'\\cmidrule(\([^)]*\))?\{[^}]*\}', '', tex)
156 +
157 +# ---------------------------------------------------------------- pandoc shims
158 +shims = r"""
159 +\newcommand{\num}[1]{#1}
160 +\renewcommand{\tablelegend}[1]{\par\textit{Notes:} #1}
161 +\renewcommand{\tstat}[1]{(\textit{t}=#1)}
162 +\newcommand{\textperiodcentered}{·}
163 +"""
164 +tex = tex.replace("\\begin{document}", shims + "\\begin{document}", 1)
165 +tex = re.sub(r'(\\newcommand\{\\WPauthor\}\{[^}]*)\}', lambda m: m.group(1).replace('\\quad', '~~~~') + '}', tex)
166 +tex = tex.replace("\\newcommand{\\textperiodcentered}{·}", "\\providecommand{\\textperiodcentered}{·}")
167 +OUT.write_text(tex)
168 +print("wrote", OUT, len(tex), "chars")
added data/amount_dist.csv +2 −0
@@ -0,0 +1,2 @@
1 +q,n,sum_gusd
2 +"[3200000.0, 23000000.0, 128500000.0, 500000000.0, 1503139999.9999979, 12500800000.000162]",56500,55910.9
added data/companies_by_sector.csv +12 −0
@@ -0,0 +1,12 @@
1 +sector,n,sp500,russell
2 +Financials,462,76,461
3 +Health Care,433,59,433
4 +Industrials,393,79,393
5 +Information Technology,333,73,330
6 +Consumer Discretionary,276,48,276
7 +Real Estate,146,31,145
8 +Materials,126,26,126
9 +Energy,119,21,118
10 +Communication Services,105,20,102
11 +Consumer Staples,103,36,103
12 +Utilities,68,31,68
added data/companies_by_shard.csv +11 −0
@@ -0,0 +1,11 @@
1 +shard_node,n
2 +M2U64,199
3 +M3U96a,194
4 +M3U96b,565
5 +M4BP36,239
6 +M4BP48,219
7 +M4M64a,197
8 +M4M64c,203
9 +M5M36,198
10 +m4mm,252
11 +m4mn,298
added data/confidence_dist.csv +7 −0
@@ -0,0 +1,7 @@
1 +conf,n
2 +0.5,1611
3 +0.6,3278
4 +0.7,21647
5 +0.8,72280
6 +0.9,100831
7 +1.0,108418
added data/duration.csv +6 −0
@@ -0,0 +1,6 @@
1 +span_years,n
2 +0,67170
3 +1-2,17307
4 +3-5,8697
5 +>5,10326
6 +unknown,17599
added data/example_provenance.csv +25 −0
@@ -0,0 +1,25 @@
1 +project_id,ticker,project_name,project_type,status,total_amount_usd,n_mentions,first_seen,last_seen,accession_number,filing_date,form_type,m_status,amount_usd,confidence,snippet
2 +00461ab6a8ee44d809b436b3aaf59072,MPT,Development Projects with First Choice Emergency Room,plant_construction,in_progress,800000000.0,8,2013-02-22,2023-03-01,0001193125-13-071643,2013-02-22,10-K,planned,800000000.0,0.8,Committed to new development projects totaling more than $800 million.
3 +00461ab6a8ee44d809b436b3aaf59072,MPT,Development Projects with First Choice Emergency Room,plant_construction,in_progress,800000000.0,8,2013-02-22,2023-03-01,0001193125-14-080657,2014-03-03,10-K,in_progress,,0.9,We have seven development projects with First Choice Emergency Room that started in 2014.
4 +00461ab6a8ee44d809b436b3aaf59072,MPT,Development Projects with First Choice Emergency Room,plant_construction,in_progress,800000000.0,8,2013-02-22,2023-03-01,0001193125-15-073689,2015-03-02,10-K,in_progress,,0.8,We have 11 development projects with First Choice Emergency Room currently under development.
5 +00461ab6a8ee44d809b436b3aaf59072,MPT,Development Projects with First Choice Emergency Room,plant_construction,in_progress,800000000.0,8,2013-02-22,2023-03-01,0001193125-16-485612,2016-02-29,10-K,in_progress,,0.8,"The company has 10 development projects under development with Adeptus Health, Inc."
6 +00461ab6a8ee44d809b436b3aaf59072,MPT,Development Projects with First Choice Emergency Room,plant_construction,in_progress,800000000.0,8,2013-02-22,2023-03-01,0001193125-18-067584,2018-03-01,10-K,in_progress,,0.8,"Three properties were under construction as of December 31, 2017."
7 +00461ab6a8ee44d809b436b3aaf59072,MPT,Development Projects with First Choice Emergency Room,plant_construction,in_progress,800000000.0,8,2013-02-22,2023-03-01,0001564590-21-009901,2021-03-01,10-K,in_progress,,0.8,"The company has two development projects still under construction at December 31, 2020."
8 +00461ab6a8ee44d809b436b3aaf59072,MPT,Development Projects with First Choice Emergency Room,plant_construction,in_progress,800000000.0,8,2013-02-22,2023-03-01,0001564590-22-008100,2022-03-01,10-K,in_progress,,0.8,"Three properties were under construction as of December 31, 2021."
9 +00461ab6a8ee44d809b436b3aaf59072,MPT,Development Projects with First Choice Emergency Room,plant_construction,in_progress,800000000.0,8,2013-02-22,2023-03-01,0000950170-23-005575,2023-03-01,10-K,in_progress,,0.8,"The portfolio includes several development projects still under construction as of December 31, 2022."
10 +3e2705ba5b2dd27408b25f0145e34250,LNG,Corpus Christi Liquefaction Stage 3 Project,plant_construction,in_progress,11500000000.0,8,2014-04-08,2022-03-07,0000003570-14-000070,2014-04-08,8-K,planned,,0.8,The construction of the first liquefaction train at the Corpus Christi Liquefaction facility is contingent upon receiving regulatory approvals and securing financing.
11 +3e2705ba5b2dd27408b25f0145e34250,LNG,Corpus Christi Liquefaction Stage 3 Project,plant_construction,in_progress,11500000000.0,8,2014-04-08,2022-03-07,0000003570-14-000114,2014-05-30,8-K,planned,,0.9,CCLNG will construct a second liquefaction train at its facility to sell LNG to IBERDROLA under a long-term agreement.
12 +3e2705ba5b2dd27408b25f0145e34250,LNG,Corpus Christi Liquefaction Stage 3 Project,plant_construction,in_progress,11500000000.0,8,2014-04-08,2022-03-07,0000003570-14-000117,2014-06-02,8-K,planned,,0.9,"Cheniere Energy, Inc. plans to construct a second liquefaction train at the Corpus Christi Liquefaction facility."
13 +3e2705ba5b2dd27408b25f0145e34250,LNG,Corpus Christi Liquefaction Stage 3 Project,plant_construction,in_progress,11500000000.0,8,2014-04-08,2022-03-07,0000003570-14-000139,2014-06-30,8-K,planned,,0.9,"Cheniere Energy, Inc. plans to construct a second liquefaction train at the Corpus Christi Liquefaction facility."
14 +3e2705ba5b2dd27408b25f0145e34250,LNG,Corpus Christi Liquefaction Stage 3 Project,plant_construction,in_progress,11500000000.0,8,2014-04-08,2022-03-07,0000003570-14-000153,2014-07-17,8-K,planned,,0.9,"Cheniere Energy, Inc. plans to construct a third liquefaction train at its Corpus Christi Liquefaction facility."
15 +3e2705ba5b2dd27408b25f0145e34250,LNG,Corpus Christi Liquefaction Stage 3 Project,plant_construction,in_progress,11500000000.0,8,2014-04-08,2022-03-07,0000003570-14-000306,2014-12-18,8-K,planned,,0.9,"Cheniere Energy, Inc. plans to construct a third liquefaction train at its Corpus Christi facility."
16 +3e2705ba5b2dd27408b25f0145e34250,LNG,Corpus Christi Liquefaction Stage 3 Project,plant_construction,in_progress,11500000000.0,8,2014-04-08,2022-03-07,0001193125-15-186045,2015-05-13,8-K,in_progress,11500000000.0,0.9,"The project involves the development, construction, operation, and maintenance of a natural gas liquefaction and export facility and related infrastructure in San Patricio County and Nueces County, Texas."
17 +3e2705ba5b2dd27408b25f0145e34250,LNG,Corpus Christi Liquefaction Stage 3 Project,plant_construction,in_progress,11500000000.0,8,2014-04-08,2022-03-07,0000003570-22-000029,2022-03-07,8-K,in_progress,5484000000.0,1.0,"The project involves the engineering, procurement, construction, pre-commissioning, commissioning, start-up, and testing of a mid-scale natural gas liquefaction facility consisting of 7 liquefaction units in San Patricio County, Texas."
18 +5317e0ee56e32e3563b0d34a8ec04442,CUZ,"Development of Domain 10, Domain 12, and Avalon 10000",plant_construction,in_progress,759000000.0,8,2014-07-29,2022-02-03,0000025232-14-000029,2014-07-29,10-Q,in_progress,,0.8,The Company has two projects (one office and one mixed use) under active development.
19 +5317e0ee56e32e3563b0d34a8ec04442,CUZ,"Development of Domain 10, Domain 12, and Avalon 10000",plant_construction,in_progress,759000000.0,8,2014-07-29,2022-02-03,0000025232-15-000006,2015-02-12,10-K,in_progress,,0.8,The Company commenced development activities on two other development projects within its target markets.
20 +5317e0ee56e32e3563b0d34a8ec04442,CUZ,"Development of Domain 10, Domain 12, and Avalon 10000",plant_construction,in_progress,759000000.0,8,2014-07-29,2022-02-03,0000025232-16-000067,2016-07-28,10-Q,in_progress,,0.7,The company has three projects (two office and one mixed-use) under active development.
21 +5317e0ee56e32e3563b0d34a8ec04442,CUZ,"Development of Domain 10, Domain 12, and Avalon 10000",plant_construction,in_progress,759000000.0,8,2014-07-29,2022-02-03,0000025232-16-000076,2016-11-01,10-Q,in_progress,,0.8,"The Company has five projects (four office and one mixed-use) under active development as of September 30, 2016."
22 +5317e0ee56e32e3563b0d34a8ec04442,CUZ,"Development of Domain 10, Domain 12, and Avalon 10000",plant_construction,in_progress,759000000.0,8,2014-07-29,2022-02-03,0000025232-19-000006,2019-02-06,10-K,in_progress,245900000.0,0.8,"In 2018, the company commenced two new development projects as part of its ongoing strategy."
23 +5317e0ee56e32e3563b0d34a8ec04442,CUZ,"Development of Domain 10, Domain 12, and Avalon 10000",plant_construction,in_progress,759000000.0,8,2014-07-29,2022-02-03,0000025232-21-000008,2021-02-11,10-K,in_progress,449400000.0,0.9,"At year-end, we had five development projects in process with a total expected cost of $449.4 million."
24 +5317e0ee56e32e3563b0d34a8ec04442,CUZ,"Development of Domain 10, Domain 12, and Avalon 10000",plant_construction,in_progress,759000000.0,8,2014-07-29,2022-02-03,0000025232-21-000030,2021-04-29,10-Q,completed,,1.0,"The completed development and commencement of operations at Domain 10, Domain 12, and Avalon 10000 contributed to increased revenues and expenses."
25 +5317e0ee56e32e3563b0d34a8ec04442,CUZ,"Development of Domain 10, Domain 12, and Avalon 10000",plant_construction,in_progress,759000000.0,8,2014-07-29,2022-02-03,0000025232-22-000005,2022-02-03,10-K,in_progress,759000000.0,0.9,"At year-end 2021, we had four development projects in process, and our share of the total expected costs of these projects totaled $759.0 million."
added data/factor_tables.csv +5 −0
@@ -0,0 +1,5 @@
1 +t,n
2 +chunk_scores,10720890
3 +filing_scores,527531
4 +annual_panel,32852
5 +quarterly_panel,128043
added data/filings_by_form.csv +4 −0
@@ -0,0 +1,4 @@
1 +form_type,n,first,last,gb
2 +10-K,31775,2010-01-08,2026-10-27,88.6
3 +10-Q,95167,2010-01-05,2026-10-27,155.9
4 +8-K,400606,2010-01-04,2026-10-27,10.0
added data/filings_by_year_form.csv +52 −0
@@ -0,0 +1,52 @@
1 +yr,form_type,n,issuers
2 +2010,10-K,1357,1355
3 +2010,10-Q,4152,1388
4 +2010,8-K,17672,1383
5 +2011,10-K,1409,1402
6 +2011,10-Q,4260,1423
7 +2011,8-K,18533,1416
8 +2012,10-K,1436,1434
9 +2012,10-Q,4363,1467
10 +2012,8-K,19437,1464
11 +2013,10-K,1483,1482
12 +2013,10-Q,4530,1528
13 +2013,8-K,19773,1534
14 +2014,10-K,1549,1548
15 +2014,10-Q,4780,1612
16 +2014,8-K,20927,1615
17 +2015,10-K,1635,1631
18 +2015,10-Q,4983,1685
19 +2015,8-K,22077,1687
20 +2016,10-K,1692,1691
21 +2016,10-Q,5146,1733
22 +2016,8-K,22447,1741
23 +2017,10-K,1745,1744
24 +2017,10-Q,5359,1805
25 +2017,8-K,22897,1807
26 +2018,10-K,1827,1825
27 +2018,10-Q,5582,1887
28 +2018,8-K,23709,1892
29 +2019,10-K,1903,1900
30 +2019,10-Q,5829,1965
31 +2019,8-K,24256,1971
32 +2020,10-K,1985,1983
33 +2020,10-Q,6095,2068
34 +2020,8-K,27424,2088
35 +2021,10-K,2116,2110
36 +2021,10-Q,6606,2240
37 +2021,8-K,27944,2246
38 +2022,10-K,2258,2256
39 +2022,10-Q,6830,2287
40 +2022,8-K,26533,2290
41 +2023,10-K,2295,2295
42 +2023,10-Q,6938,2326
43 +2023,8-K,26851,2329
44 +2024,10-K,2341,2340
45 +2024,10-Q,7092,2382
46 +2024,8-K,27438,2386
47 +2025,10-K,2406,2405
48 +2025,10-Q,7300,2448
49 +2025,8-K,28716,2453
50 +2026,10-K,2338,2337
51 +2026,10-Q,5322,2493
52 +2026,8-K,23972,2498
added data/issuers_by_year.csv +18 −0
@@ -0,0 +1,18 @@
1 +yr,issuers,filings
2 +2010,1392,23181
3 +2011,1424,24202
4 +2012,1472,25236
5 +2013,1539,25786
6 +2014,1619,27256
7 +2015,1689,28695
8 +2016,1744,29285
9 +2017,1810,30001
10 +2018,1893,31118
11 +2019,1973,31988
12 +2020,2089,35504
13 +2021,2247,36666
14 +2022,2291,35621
15 +2023,2329,36084
16 +2024,2387,36871
17 +2025,2453,38422
18 +2026,2498,31632
added data/issuers_without_filings.csv +66 −0
@@ -0,0 +1,66 @@
1 +ticker,company_name,sector,in_sp500,in_russell3000
2 +ALM,ALMONTY INDUSTRIES,Materials,False,True
3 +AMBP,ARDAGH METAL PACKAGING SA,Materials,False,True
4 +AS,AMER SPORTS INC,Consumer Discretionary,False,True
5 +ASC,ARDMORE SHIPPING CORP,Energy,False,True
6 +ASND,ASCENDIS PHARMA,Health Care,False,True
7 +AU,ANGLOGOLD ASHANTI PLC,Materials,False,True
8 +AUGO,AURA MINERALS INC,Materials,False,True
9 +BBUC,BROOKFIELD BUSINESS CORP CLASS A,Industrials,False,True
10 +BEPC,BROOKFIELD RENEWABLE SUBORDINATE V,Utilities,False,True
11 +BIPC,BROOKFIELD INFRASTRUCTURE CORP CLA,Utilities,False,True
12 +BIRK,BIRKENSTOCK HOLDING PLC,Consumer Discretionary,False,True
13 +BLSH,BULLISH,Financials,False,True
14 +BLX,BLADEX INC CLASS E,Financials,False,True
15 +BORR,BORR DRILLING,Energy,False,True
16 +BRSL,BRIGHTSTAR LOTTERY PLC,Consumer Discretionary,False,True
17 +BTDR,BITDEER TECHNOLOGIES GROUP CLASS A,Information Technology,False,True
18 +BULL,WEBULL CORP CLASS A,Financials,False,True
19 +CMCL,CALEDONIA MINING PLC,Materials,False,True
20 +CMDB,COSTAMARE BULKERS HOLDINGS LTD,Industrials,False,True
21 +CRML,CRITICAL METALS CORP,Materials,False,True
22 +CSIQ,CANADIAN SOLAR,Information Technology,False,True
23 +DHT,DHT HOLDINGS,Energy,False,True
24 +DLO,DLOCAL CLASS A,Financials,False,True
25 +DOX,AMDOCS,Information Technology,False,True
26 +DRUG,BRIGHT MINDS BIOSCIENCES,Health Care,False,True
27 +ELE,ELEMENTAL ROYALTY CORP,Materials,False,True
28 +FLNG,FLEX LNG,Energy,False,True
29 +GENI,GENIUS SPORTS LTD,Consumer Discretionary,False,True
30 +GFL,GFL ENVIRONMENTAL SUBORDINATE VOTI,Industrials,False,True
31 +GFS,GLOBALFOUNDRIES,Information Technology,False,True
32 +GLNG,GOLAR LNG,Energy,False,True
33 +GLOB,GLOBANT,Information Technology,False,True
34 +HSHP,HIMALAYA SHIPPING LTD,Industrials,False,True
35 +LZM,LIFEZONE METALS LTD,Materials,False,True
36 +MDWD,MEDIWOUND LTD,Health Care,False,True
37 +MRX,MAREX GROUP LTD,Financials,False,True
38 +NAT,NORDIC AMERICAN TANKERS,Energy,False,True
39 +NEXN,NEXXEN INTERNATIONAL LTD,Communication Services,False,True
40 +NTB,BANK OF NT BUTTERFIELD & SON LTD,Financials,False,True
41 +NU,NU HOLDINGS CLASS A,Financials,False,True
42 +NVGS,NAVIGATOR HOLDINGS,Energy,False,True
43 +ONON,ON HOLDING AG CLASS A,Consumer Discretionary,False,True
44 +OZK,BANK OZK,Financials,False,True
45 +PAGS,PAGSEGURO DIGITAL LTD CLASS A,Financials,False,True
46 +PAX,PATRIA INVESTMENTS LTD CLASS A,Financials,False,True
47 +PFBC,PREFERRED BANK,Financials,False,True
48 +PLGO,PELAGOS INSURANCE CAPITAL LTD,Financials,False,True
49 +PSFE,PAYSAFE LTD,Financials,False,True
50 +QGEN,QIAGEN,Health Care,False,True
51 +SB,SAFE BULKERS,Industrials,False,True
52 +SFL,SFL,Energy,False,True
53 +SGHC,SUPER GROUP,Consumer Discretionary,False,True
54 +SPOT,SPOTIFY TECHNOLOGY,Communication Services,False,True
55 +STNE,STONECO CLASS A,Financials,False,True
56 +STNG,SCORPIO TANKERS,Energy,False,True
57 +TATT,TAT TECHNOLOGIES LTD,Industrials,False,True
58 +TIGO,MILLICOM INTERNATIONAL CELLULAR,Communication Services,False,True
59 +TK,TEEKAY CORPORATION CORP LTD,Energy,False,True
60 +TNK,TEEKAY TANKERS LTD CLASS A,Energy,False,True
61 +TTAM,TITAN AMERICA SA,Materials,False,True
62 +VIK,VIKING HOLDINGS LTD,Consumer Discretionary,False,True
63 +VUECF,VERDERA ENERGY CORP,Energy,False,True
64 +WSHP,WESHOP HOLDINGS CLASS A,Communication Services,False,True
65 +XP,XP CLASS A INC,Financials,False,True
66 +ZGN,ERMENEGILDO ZEGNA NV,Consumer Discretionary,False,True
added data/kg.csv +9 −0
@@ -0,0 +1,9 @@
1 +k,node_type,n
2 +edges,owns,121099
3 +edges,located_in,54583
4 +edges,uses,28774
5 +nodes,Project,121099
6 +nodes,Partner,36822
7 +nodes,Location,14508
8 +nodes,Technology,8551
9 +nodes,Company,2492
added data/ledger_10k_items.csv +7 −0
@@ -0,0 +1,7 @@
1 +section_name,n,mentions
2 +business,29285,30492
3 +risk_factors,29158,2797
4 +md_and_a,28919,39099
5 +properties,28236,7879
6 +quantitative_disclosures,26876,531
7 +full_body,1230,824
added data/ledger_10q_items.csv +9 −0
@@ -0,0 +1,9 @@
1 +section_name,n,mentions
2 +controls_and_procedures,92393,7082
3 +financial_statements,91882,30159
4 +md_and_a,91371,99305
5 +quantitative_disclosures,83138,808
6 +unregistered_sales,80210,9535
7 +risk_factors,72560,4420
8 +legal_proceedings,66998,845
9 +full_document,241,107
added data/ledger_by_form.csv +4 −0
@@ -0,0 +1,4 @@
1 +form_type,sections,with_mentions,mentions,issuers
2 +10-K,143704,30877,81622,2464
3 +10-Q,578793,75705,152261,2494
4 +8-K,400601,61648,74594,2499
added data/mentions_by_form.csv +4 −0
@@ -0,0 +1,4 @@
1 +form_type,n,mean_conf
2 +10-Q,152004,0.885
3 +10-K,81495,0.878
4 +8-K,74566,0.922
added data/mentions_by_year.csv +18 −0
@@ -0,0 +1,18 @@
1 +yr,mentions,issuers
2 +2010,11908,1224
3 +2011,13551,1287
4 +2012,14302,1333
5 +2013,14611,1408
6 +2014,16274,1490
7 +2015,17073,1535
8 +2016,16429,1579
9 +2017,17350,1636
10 +2018,18064,1737
11 +2019,18737,1806
12 +2020,17970,1919
13 +2021,21312,2077
14 +2022,22720,2105
15 +2023,22955,2129
16 +2024,22503,2204
17 +2025,23220,2280
18 +2026,19086,2260
added data/mentions_per_project.csv +6 −0
@@ -0,0 +1,6 @@
1 +bucket,n
2 +1,72932
3 +2,20544
4 +3-5,17681
5 +6-10,6173
6 +>10,3769
added data/parse_status.csv +9 −0
@@ -0,0 +1,9 @@
1 +form_type,parse_status,n
2 +10-K,complete,31772
3 +10-K,error,2
4 +10-K,missing,1
5 +10-Q,complete,95155
6 +10-Q,error,11
7 +10-Q,missing,1
8 +8-K,complete,400605
9 +8-K,error,1
added data/projects_by_first_seen_year.csv +18 −0
@@ -0,0 +1,18 @@
1 +yr,n
2 +2010,7835
3 +2011,6679
4 +2012,6444
5 +2013,6286
6 +2014,6732
7 +2015,6675
8 +2016,6240
9 +2017,6490
10 +2018,6815
11 +2019,6837
12 +2020,7212
13 +2021,8216
14 +2022,8158
15 +2023,7977
16 +2024,7819
17 +2025,7990
18 +2026,6694
added data/projects_by_index.csv +4 −0
@@ -0,0 +1,4 @@
1 +in_sp500,in_russell3000,projects,issuers
2 +False,True,93333,1992
3 +True,True,27382,491
4 +True,False,384,9
added data/projects_by_sector.csv +12 −0
@@ -0,0 +1,12 @@
1 +sector,n,issuers,per_issuer
2 +Health Care,23679,429,55.2
3 +Industrials,17890,386,46.3
4 +Consumer Discretionary,15072,267,56.4
5 +Financials,14223,446,31.9
6 +Information Technology,11756,325,36.2
7 +Real Estate,9001,145,62.1
8 +Materials,7407,117,63.3
9 +Energy,6710,107,62.7
10 +Consumer Staples,5921,103,57.5
11 +Utilities,4920,66,74.5
12 +Communication Services,4520,101,44.8
added data/projects_by_status.csv +5 −0
@@ -0,0 +1,5 @@
1 +status,n,pct
2 +in_progress,50327,41.6
3 +completed,49687,41.0
4 +planned,18340,15.1
5 +mentioned,2745,2.3
added data/projects_by_type.csv +22 −0
@@ -0,0 +1,22 @@
1 +project_type,n,pct,with_amount,median_amount_musd,mean_mentions
2 +ma_integration,41742,34.5,28044,155.0,3.12
3 +capex_program,10720,8.9,9452,300.0,1.79
4 +partnership,9909,8.2,2943,75.0,1.77
5 +geographic_expansion,7868,6.5,1431,36.7,2.25
6 +plant_construction,7590,6.3,2511,56.0,2.42
7 +cost_reduction,6481,5.4,4378,50.0,2.03
8 +drug_pipeline,5360,4.4,778,46.1,4.99
9 +product_launch,5356,4.4,447,26.7,2.33
10 +rd_program,5080,4.2,1272,20.0,2.31
11 +infrastructure,4481,3.7,1694,48.0,1.9
12 +manufacturing_expansion,4140,3.4,1040,45.0,2.13
13 +technology_deployment,4128,3.4,1219,135.0,1.63
14 +digital_transformation,2642,2.2,240,34.2,1.99
15 +erp_implementation,1735,1.4,184,15.0,2.44
16 +sustainability,1305,1.1,410,34.4,2.0
17 +supply_chain,684,0.6,139,38.0,1.71
18 +data_center,449,0.4,129,166.0,1.98
19 +energy_transition,415,0.3,109,109.0,2.09
20 +ai_initiative,376,0.3,20,127.5,1.68
21 +automation,374,0.3,25,13.7,1.68
22 +cloud_migration,264,0.2,35,50.0,1.97
added data/projects_per_issuer.csv +2 −0
@@ -0,0 +1,2 @@
1 +q,mn,mx,mean
2 +"[12.0, 24.0, 42.0, 67.0, 94.0, 140.0]",1,240,48.59510433386838
added data/sections_10k_by_item.csv +16 −0
@@ -0,0 +1,16 @@
1 +section_name,n,median_words
2 +market_for_equity,31094,568.0
3 +md_and_a,31000,10138.0
4 +controls_and_procedures,30917,597.0
5 +business,30738,5346.0
6 +risk_factors,30685,9353.0
7 +properties,30241,200.0
8 +financial_statements,30096,16452.0
9 +quantitative_disclosures,30025,502.0
10 +legal_proceedings,29051,84.0
11 +other_information,26506,2731.0
12 +selected_financial_data,19509,524.0
13 +changes_in_accounting,12628,2842.0
14 +mine_safety,6240,533.0
15 +unresolved_staff_comments,2391,53.0
16 +full_document,86,72521.0
added data/sections_by_form.csv +4 −0
@@ -0,0 +1,4 @@
1 +form_type,sections,mean_words,median_words
2 +10-K,341207,5551.0,875.0
3 +10-Q,600471,3843.0,500.0
4 +8-K,400605,769.0,575.0
added data/stats.json +74 −0
@@ -0,0 +1,74 @@
1 +{
2 + "universe": {
3 + "built_at": "2026-10-27T16:14:42.716940+00:00",
4 + "n": 2564,
5 + "n_sp500": 500,
6 + "n_russell": 2555,
7 + "russell_holdings_as_of": "Oct 23, 2026",
8 + "russell_fetched_at": "2026-10-27T16:14:42.459987+00:00",
9 + "russell_equity_lines_mapped": 2573,
10 + "russell_unmatched": 7,
11 + "russell_unmatched_list": [
12 + "TOWN (TOWNEBANK)",
13 + "FRBA (FIRST BANK)",
14 + "NBN (NORTHEAST BANK)",
15 + "HIFS (HINGHAM INSTITUTION FOR SAVINGS)",
16 + "ADRO (CHINOOK THERAPEUTICS INC)",
17 + "THRD (THIRD HARMONIC BIO INC)",
18 + "HOLX (HOLOGIC INC)"
19 + ],
20 + "sp500_fetched_at": "2026-05-25T16:15:58.318619+00:00",
21 + "sp500_rows": 503,
22 + "russell_distinct_ciks": 2555,
23 + "both": 491,
24 + "sp_only": 9,
25 + "ru_only": 2064
26 + },
27 + "issuers_with_filings": 2499,
28 + "issuers_without_filings": 65,
29 + "issuers_with_projects": 2492,
30 + "chunks": [
31 + 10720890,
32 + 479.7
33 + ],
34 + "embeddings": 10720890,
35 + "sections_total": 1342283,
36 + "filings_total": 527548,
37 + "raw_gb": 254.5,
38 + "ledger_total": 1123098,
39 + "mentions": 308065,
40 + "projects": 121099,
41 + "mentions_fields": [
42 + 120240,
43 + 118783,
44 + 36740,
45 + 81320,
46 + 252469,
47 + 308065
48 + ],
49 + "projects_fields": [
50 + 56500,
51 + 54583,
52 + 17694,
53 + 121099
54 + ],
55 + "spid_runs": {
56 + "sp500_run": "2026-06 (gpt-4o-mini, 16 workers, 153,894 sections)",
57 + "russell_run": "2026-10-27/28 (gpt-4o-mini-2024-07-18, 48 workers, 968,134 sections)"
58 + },
59 + "extracted_at_range": [
60 + "2026-10-27 13:12:32.140400",
61 + "2026-10-28 11:15:49.150818"
62 + ],
63 + "example_project": {
64 + "project_id": "3e2705ba5b2dd27408b25f0145e34250",
65 + "ticker": "LNG",
66 + "project_name": "Corpus Christi Liquefaction Stage 3 Project",
67 + "project_type": "plant_construction",
68 + "status": "in_progress",
69 + "total_amount_usd": "11500000000.0",
70 + "n_mentions": "8",
71 + "first_seen": "2014-04-08",
72 + "last_seen": "2022-03-07"
73 + }
74 +}
\ No newline at end of file
added data/top_issuers.csv +16 −0
@@ -0,0 +1,16 @@
1 +ticker,company_name,sector,projects
2 +UEC,URANIUM ENERGY,Energy,240
3 +UUUU,ENERGY FUELS,Energy,214
4 +JBLU,JETBLUE AIRWAYS,Industrials,203
5 +HAL,Halliburton,Energy,201
6 +RIOT,RIOT PLATFORMS,Information Technology,195
7 +EHC,ENCOMPASS HEALTH,Health Care,191
8 +EBS,EMERGENT BIOSOLUTIONS,Health Care,187
9 +MUX,MCEWEN INC,Materials,180
10 +CDE,COEUR MINING,Materials,180
11 +CLSK,CLEANSPARK,Information Technology,170
12 +CWEN,CLEARWAY ENERGY INC CLASS C,Utilities,168
13 +ALK,ALASKA AIR GROUP INC,Industrials,168
14 +PENN,PENN ENTERTAINMENT,Consumer Discretionary,159
15 +EOG,EOG Resources,Energy,159
16 +APLE,APPLE HOSPITALITY REIT INC,Real Estate,155
added docs/contributions.md +15 −0
@@ -0,0 +1,15 @@
1 +# Contributions (CRediT) — to be confirmed by all coauthors before circulation
2 +
3 +Author order used in the draft (UQO convention of the team's earlier WP 17): **Manel Kammoun, Djerrycharli (Charlie) Tandjambianda, Simon-Pierre Boucher**. The project brief proposed *Tandjambianda, Boucher, Kammoun*. **Open decision — the team must choose.** Affiliations, ORCID and corresponding author also to be confirmed.
4 +
5 +| Contribution (CRediT) | Kammoun | Tandjambianda | Boucher |
6 +|---|---|---|---|
7 +| Conceptualization | to confirm | to confirm | to confirm |
8 +| Methodology (EDGAR corpus, SPID pipeline) | to confirm | to confirm | yes (code, Oct. 2026 extension) |
9 +| Software / data curation (Russell 3000 build) | — | — | yes |
10 +| Validation design | to confirm | to confirm | yes (protocol) |
11 +| Writing – original draft (WP 28) | — | — | yes |
12 +| Writing – review & editing | to confirm | to confirm | yes |
13 +| Funding / supervision | to confirm | to confirm | to confirm |
14 +
15 +Nothing in this table attributes a role that has not been explicitly confirmed; cells marked "to confirm" are placeholders, not claims. The June 2026 methodological study (Tandjambianda, Boucher, Kammoun) and the S&P 500 release are prior joint work of the three authors.
added docs/data_dictionary.md +25 −0
@@ -0,0 +1,25 @@
1 +# Data dictionary (current tables, revision 2324b58)
2 +
3 +## Corpus layer — `data/duckdb/edgar_research.duckdb` (68 GB) and `data/parquet/*`
4 +| Table | Key | Fields |
5 +|---|---|---|
6 +| `companies` | `cik` | ticker, company_name, sector, sub_industry, sic_code, sic_description, state, state_of_incorporation, exchange, fiscal_year_end, date_added, in_sp500, in_russell3000, russell_weight_pct, shard_node |
7 +| `filings` | `accession_number` | cik (FK), ticker, company_name, form_type, filing_date, report_date, fiscal_year, sec_url, local_path, primary_document, file_size_bytes, download_status, parse_status (complete/error/missing) |
8 +| `filing_sections` | `section_id` (UUID) | accession_number (FK), section_name, item_number, section_order, text, word_count, char_count |
9 +| `filing_chunks` | `chunk_id` (UUID) | section_id (FK), accession_number, chunk_text, chunk_order, word_count, char_count, start_char, end_char (offsets within the section) |
10 +| `embeddings_metadata` / `parquet/embeddings/` | `embedding_id` | source_id = chunk_id, model all-MiniLM-L6-v2, 384-d float32 vectors in Parquet |
11 +| `chunk_scores`, `filing_scores`, `quarterly_panel`, `annual_panel`, `*_final` | chunk / accession / firm-period | cosine scores to 8 thematic anchors and their aggregates |
12 +
13 +## Project layer — `project-db/data/db/spid.duckdb` (3.2 GB)
14 +| Table | Key | Fields |
15 +|---|---|---|
16 +| `extracted_sections` | `section_id` | accession_number, backend, n_mentions, extracted_at (ledger of every section submitted) |
17 +| `spid_sections` | `section_id` | legacy 10-K items ingested in June 2026 (27,164 rows) |
18 +| `project_mentions` | `mention_id` | accession_number, cik, section_id, section_name, project_name, project_type, description, objective, amount_usd, amount_raw, locations[], technologies[], partners[], suppliers[], benefits[], risks[], years[], status, confidence, backend, ticker, company_name, form_type, filing_date, fiscal_year, sector |
19 +| `projects` | `project_id` | cik, ticker, company_name, sector, project_type, project_name, canonical_location, technologies[], status, total_amount_usd (max), n_mentions, n_filings, first_seen, last_seen, first_year, last_year, avg_confidence, description |
20 +| `project_timeline` | (project_id, accession_number) | filing_date, form_type, status, amount_usd, confidence, snippet |
21 +| `kg_nodes` / `kg_edges` | node_id / (src, rel, dst) | node_type ∈ {Company, Project, Location, Technology, Partner}; rel ∈ {owns, located_in, uses} |
22 +| `project_embeddings` | `project_id` | model, dim, vector FLOAT[384] |
23 +| views `src_filings`, `src_sections`, `src_companies`, `v_sections` | — | read-only views over the corpus Parquet (paths of the master node) |
24 +
25 +Null conventions: unknown amounts/locations are NULL / empty lists; `fiscal_year` is NULL for mentions of the June 2026 run.
added docs/implementation_audit.md +33 −0
@@ -0,0 +1,33 @@
1 +# Implementation audit — UQO WP 28 (checked-out revision 2324b58, 2026-10-28)
2 +
3 +Repository: `gitsrv:srv/git/uqo-research-db.git` (private UQO Git server; the public alias cited in the brief, `git.spboucher.ai/uqo-research-db.git`, points to the same server). The June 2026 methodological study examined `df7905d`; the paper's numbers come from the build at `2324b58`.
4 +
5 +## Inspected (paths as checked out)
6 +- `src/acquisition/{universe,bulk_downloader,cik_mapper,sec_client}.py`, `scripts/{build_universe,download_filings,build_database,parse_filings,build_embeddings,build_factors,export_shard,merge_shards}.py`, `scripts/cluster/{orchestrate.py,rebalance.py,run_shard.sh,spid_run.sh}`
7 +- `project-db/spid/{taxonomy,pipeline,resolve,graph,embed,store,config}.py`, `project-db/spid/extract/{base,rules,llm}.py`, `project-db/run_full_extract.py`
8 +- Artifacts: `data/metadata/universe.json`, `shards.json`, `download_log*.jsonl`, the two DuckDB files, `data/logs/spid_extract.log` (outage window 2026-10-28 01:10 → 06:22)
9 +- No `AGENTS.md`; `CLAUDE.md` files exist only at the user level. No unit tests exist for the SPID package.
10 +
11 +## Evidence-backed discrepancies vs. the brief
12 +| # | Finding | Evidence | Priority |
13 +|---|---|---|---|
14 +| 1 | No human-labelled gold standard; precision/recall unknown | no `human_labels` table, no annotation files | High |
15 +| 2 | Resolution key merges distinct phases / splits renamed projects; no link scores | `resolve.py` (MD5 of cik|type|loc/token); Cheniere example | High |
16 +| 3 | `extracted_sections` cannot distinguish "no project" from "API failure" | 149,352 zero-mention rows written during the credit outage, removed by timestamp | High |
17 +| 4 | Amounts untyped; project amount = max over mentions | `resolve.py`; `amount_raw` kept | High |
18 +| 5 | No `project_events`; timeline = mention rows; cited years unordered | `project_timeline` schema | Medium |
19 +| 6 | No prompt / taxonomy / schema hash per run | `extracted_sections.backend` = "llm" only | Medium |
20 +| 7 | Evidence = model paraphrase; no character span of the supporting passage | `project_mentions.description` | Medium |
21 +| 8 | Focus windows truncate long sections (budget ≈ 9–12.6k chars) — recall on 10-K MD&A unknown | `llm.py::_focus` | Medium |
22 +| 9 | Universe: no file SHA-256, no match confidence, no membership dates, no security-vs-issuer table | `universe.json` | Medium |
23 +| 10 | Form scope excludes 8-K exhibits (press releases) and 20-F/40-F (65 issuers) | `download_filings.py` | Medium |
24 +| 11 | Sections lack offsets into raw HTML (chunks have offsets within sections) | `filing_sections` schema | Low |
25 +| 12 | `pq.write_to_dataset(overwrite_or_ignore)` leaves stale partitions → duplicates in `filing_scores` until cleaned (done 2026-10-28) | `aggregation.py` | Low |
26 +| 13 | Partner nodes of the knowledge graph have no edges | `graph.py` | Low |
27 +
28 +## Prioritised plan
29 +1. Annotation study (protocol in paper §8): sample, two annotators, adjudication, locked holdout → metrics.
30 +2. Schema migrations (additive): `mention_label`, `evidence_span`, `call_status` in the ledger, `extraction_runs` with hashes, `project_mention_links`, `project_events`, `project_amounts`, `human_labels`.
31 +3. Probabilistic linker with reviewable clusters; re-derive `projects` from links.
32 +4. Second extraction pass for `amount_type`; 8-K exhibit 99 ingestion; recall test on long MD&As with larger focus budgets.
33 +5. Universe hardening (hashes, confidence, security table); evaluate a licensed historical constituent list.
added docs/project_ontology.md +28 −0
@@ -0,0 +1,28 @@
1 +# Project ontology (SPID, revision 2324b58)
2 +
3 +## Units
4 +- **project_mention** — one sourced statement about an initiative in one section of one filing (table `project_mentions`; id = SHA-1(accession | section_id | type | name)).
5 +- **project** — a resolved initiative, one or more mentions (table `projects`; id = MD5(cik | project_type | first location or first ≥4-letter name token or "general")).
6 +
7 +## Labels (intended five-way classification)
8 +| Label | Definition | Current implementation |
9 +|---|---|---|
10 +| `project` | distinct, identifiable, issuer-associated strategic initiative (planned/announced/ongoing/completed/postponed/cancelled) | stored |
11 +| `generic_strategy` | non-specific aspirations ("we invest in innovation") | excluded by prompt rule + confidence ≥ 0.45; not stored |
12 +| `historical_or_third_party_reference` | initiative not attributable to the sampled issuer | not stored (prompt instruction only) |
13 +| `uncertain_candidate` | plausible initiative without identity evidence | not stored (confidence only) |
14 +| `none` | no initiative | empty extraction |
15 +
16 +Proposed change: add `mention_label` (five values) and `evidence_span` (character offsets in the section) to `project_mentions`; filled by annotators first, then by a classifier evaluated on the gold standard.
17 +
18 +## Taxonomy
19 +21 categories, unchanged since June 2026 (`project-db/spid/taxonomy.py`): manufacturing_expansion, plant_construction, data_center, cloud_migration, ai_initiative, digital_transformation, erp_implementation, rd_program, product_launch, drug_pipeline, ma_integration, geographic_expansion, capex_program, supply_chain, automation, sustainability, energy_transition, infrastructure, cost_reduction, partnership, technology_deployment. Any extension must be versioned with a crosswalk (old code → new code).
20 +
21 +## Status vocabulary
22 +`planned | in_progress | completed | mentioned` (per mention; project status = latest mention by filing date). Proposed: dated `project_events` (announcement, filing/acceptance, expected start, observed start, expected completion, realized completion, postponement, cancellation).
23 +
24 +## Amounts
25 +`amount_usd` per mention (coerced float, raw string kept); project `total_amount_usd` = **max** over mentions. Never sum across projects; never read as realized expenditure. Proposed `project_amounts` with `amount_type` ∈ {announced_budget, estimated_cost, capex_incurred, transaction_value, contract_value, financing, other}, currency, scale, basis, estimate/realized flag.
26 +
27 +## Known failure modes of the resolution key
28 +False merges (same type + same location/token: multi-phase sites, serial acquirers in one city, drug programs sharing a token); false splits (renamed initiatives, location granularity, category change); no cross-issuer identity (JVs counted twice; inherited projects restart).
added docs/research_log.md +16 −0
@@ -0,0 +1,16 @@
1 +# Research log — Russell 3000 strategic project database
2 +
3 +| Date | Decision / event | Owner | Status |
4 +|---|---|---|---|
5 +| 2026-05-25 | S&P 500 universe cached (Wikipedia table) | team | done |
6 +| 2026-06 | SPID release 1: 500 issuers, 153,894 sections, 39,930 mentions, 19,227 projects (gpt-4o-mini) | team | done |
7 +| 2026-10-27 | Extension to Russell 3000 requested; universe = iShares IWV holdings (23 Oct 2026) as proxy — **decision: snapshot estimand, no historical membership** | SPB | done |
8 +| 2026-10-27 | Distributed build (10 cluster nodes + 1 external gateway); contiguous CIK shards rebalanced after 1 h (interleaving) | SPB | done |
9 +| 2026-10-27/28 | Full LLM extraction of 968,134 new sections (48 workers), ≈ 775 USD total | SPB (authorised by SPB) | done |
10 +| 2026-10-28 01:10–06:22 | OpenAI credit exhaustion → 149,352 ledger rows with 0 mentions removed and re-run; **decision: add `call_status` to the ledger** | SPB | rows fixed; schema change proposed |
11 +| 2026-10-28 | Merge, factors, pdb-api redeploy, French methodological report updated, WP 28 draft (English) | SPB | done |
12 +| open | Author order and CRediT confirmation | all | **pending** |
13 +| open | Annotation study (sample design in WP 28 §8) | all | **pending** |
14 +| open | Linkage audit; probabilistic linker | to assign | pending |
15 +| open | Amount typing pass; 8-K exhibit ingestion; recall test on long MD&A | to assign | pending |
16 +| open | Licensed Russell constituent history? | to assign | pending |
added docs/universe_definition.md +16 −0
@@ -0,0 +1,16 @@
1 +# Universe definition (frozen for UQO WP 28, build of 2026-10-27/28)
2 +
3 +**Estimand.** Current constituents at a snapshot date, approximated by fund holdings. Not historical point-in-time Russell 3000 membership (no licensed constituent history was available).
4 +
5 +| Item | Value |
6 +|---|---|
7 +| Primary source | iShares Russell 3000 ETF (IWV) `latest-holdings.csv`, "Fund Holdings as of Oct 23, 2026", fetched 2026-10-27 (`data/metadata/russell3000_constituents.json`: `source`, `holdings_as_of`, `fetched_at`) |
8 +| Secondary source | S&P 500 constituent table (Wikipedia), cached 2026-05-25 (`sp500_constituents.json`, 503 rows / 500 CIKs) |
9 +| Identifier mapping | ticker → CIK via SEC `company_tickers.json` (fetched 2026-10-27); tickers normalised (`BRK B` → `BRK-B`) |
10 +| Exclusions | asset class ≠ Equity; CVR / escrow / rights lines; tickers with no SEC CIK (FDIC-filing banks). The 21 unmatched lines are listed in the `unmatched` field of the artifact. |
11 +| Result | 2,587 equity lines → 2,555 CIKs (share classes collapsed) → merged with S&P 500 → **2,564 issuers** (491 in both, 9 S&P-only, 2,064 Russell-only) |
12 +| Filing scope | Forms 10-K, 10-Q, 8-K (primary document only, no amendments, no exhibits), filed 2010-01-01 → 2026-10-27 |
13 +| Observed | 2,499 issuers with ≥1 filing; 65 without (foreign private issuers filing 20-F/40-F; FDIC banks) — `data/issuers_without_filings.csv` |
14 +| Frozen artifact | `data/metadata/universe.json` (`built_at` 2026-10-27), copied to the paper folder under `data/` |
15 +
16 +**Open items (not implemented):** SHA-256 of the source files; per-row match confidence; membership dates; security-vs-issuer table (`universe_snapshots`); resolution of ticker changes and spin-offs over the sample period; a licensed historical constituent list.
added paper/.latexmkrc +5 −0
@@ -0,0 +1,5 @@
1 +$pdf_mode = 1;
2 +$pdflatex = 'pdflatex -interaction=nonstopmode -halt-on-error -synctex=1 %O %S';
3 +$bibtex_use = 2;
4 +$clean_ext = 'synctex.gz run.xml bbl bcf fdb_latexmk fls log aux out toc lof lot blg';
5 +@default_files = ('main.tex');
added paper/Makefile +11 −0
@@ -0,0 +1,11 @@
1 +.PHONY: all clean distclean
2 +
3 +all:
4 + latexmk
5 +
6 +clean:
7 + latexmk -c
8 +
9 +distclean:
10 + latexmk -C
11 + rm -f *.synctex.gz *.run.xml *.bbl *.bcf
added paper/appendix/appendix.tex +110 −0
@@ -0,0 +1,110 @@
1 +% ============================================================================
2 +\begin{appendices}
3 +% ============================================================================
4 +
5 +\section{Taxonomy}\label{app:taxonomy}
6 +
7 +\begin{table}[H]
8 +\centering\small
9 +\caption{The 21-category SPID project taxonomy (\texttt{spid/taxonomy.py}, unchanged since the June 2026 release)}\label{tab:taxonomy}
10 +\input{tables/tab_taxonomy}
11 +\end{table}
12 +
13 +\section{Extraction contract}\label{app:prompt}
14 +
15 +The system prompt below is passed verbatim to the chat-completion endpoint, with the 21 taxonomy codes and labels substituted at the placeholder. The user message is the string ``\texttt{SECTION: <section\_name>}'' followed by the focused section text, truncated to 14{,}000 characters. Decoding parameters: temperature 0, JSON-object response format, 2{,}000 completion tokens. Model identifier recorded during the October 2026 run: \texttt{gpt-4o-mini-2024-07-18}.
16 +
17 +\input{tables/prompt}
18 +
19 +\noindent The candidate gate is the case-insensitive regular expression \texttt{\textbackslash b(project|initiative|program|invest|expan|construct|build|launch|develop|deploy|implement|acqui|partner|facilit|capacity|capital|migrat|transform|pipeline|plant|data ?cent|automat|sustainab|renewable|restructur)\textbackslash w*}. Sections without a match, or with fewer than 30 words, are not submitted.
20 +
21 +\section{Data dictionary and data-model audit}\label{app:dictionary}
22 +
23 +\begin{table}[H]
24 +\centering\footnotesize
25 +\caption{Tables required by the research design, current implementation and status}\label{tab:datamodel}
26 +\begin{tabularx}{\textwidth}{lXl}
27 +\toprule
28 +Required table & Current implementation (database, table, key fields) & Status \\
29 +\midrule
30 +\texttt{universe\_snapshots} & \texttt{data/metadata/universe.json} + \texttt{companies} (CIK, ticker(s), name, sector, flags \texttt{in\_sp500}/\texttt{in\_russell3000}, fund weight, SIC, state, exchange, FYE, shard); snapshot date and source URL recorded & Partial: no file hash, no membership dates, no match confidence \\
31 +\texttt{issuers} & \texttt{companies} (one row per CIK; former names from SEC header stored in metadata JSON only) & Partial: no name history, no parent/subsidiary links \\
32 +\texttt{filings} & \texttt{filings} (accession PK, CIK FK, form, filing/report dates, SEC URL, local path, size, download and parse status) & Implemented; no acceptance timestamp, no checksum \\
33 +\texttt{sections} & \texttt{filing\_sections} (UUID PK, accession FK, name, item, order, text, counts) & Implemented; no character offsets into raw HTML \\
34 +\texttt{project\_mentions} & \texttt{project\_mentions} (SHA-1 id, accession, CIK, section id, name, type, description, objective, amount, raw amount, lists, years, status, confidence, backend, issuer context) & Implemented; no evidence span, no prompt hash, no five-way label \\
35 +\texttt{projects} & \texttt{projects} (MD5 of CIK$|$type$|$location-or-name, representative fields, max amount, counts, first/last seen and cited years) & Implemented (deterministic key) \\
36 +\texttt{project\_mention\_links} & implicit in the key (every mention maps to exactly one project) & Proposed: explicit table with score, resolver version, adjudication \\
37 +\texttt{project\_events} & \texttt{project\_timeline} (one row per mention: date, form, status, amount, confidence, snippet) & Partial: mention rows, not typed events \\
38 +\texttt{project\_amounts} & \texttt{amount\_usd}/\texttt{amount\_raw} on mentions; max on projects & Proposed: type, currency, scale, basis, realization flag \\
39 +\texttt{extraction\_runs} & \texttt{extracted\_sections} ledger (section, accession, backend, n\_mentions, timestamp) + run logs & Partial: no prompt/schema/taxonomy hashes, no call status \\
40 +\texttt{human\_labels} & --- & Proposed \\
41 +\bottomrule
42 +\end{tabularx}
43 +\end{table}
44 +
45 +\noindent The full field-level dictionary of the current tables is maintained in \texttt{docs/data\_dictionary.md} of the working-paper folder and in the repository documentation.
46 +
47 +\section{Evidence register}\label{app:evidence}
48 +
49 +\begin{table}[H]
50 +\centering\footnotesize
51 +\caption{Evidence register: status of the main claims of this paper at revision \texttt{2324b58}}\label{tab:evidence}
52 +\begin{tabularx}{\textwidth}{Xl}
53 +\toprule
54 +Claim & Status \\
55 +\midrule
56 +Universe of 2{,}564 issuers built from the IWV holdings file (2026-10-23) and the S\&P~500 cache (2026-05-25) & Implemented, verified (artifact counts) \\
57 +527{,}548 primary documents of Forms 10-K/10-Q/8-K since 2010 downloaded; 16 unparsed & Implemented, verified (database counts) \\
58 +Acquisition respected the SEC rate limit (aggregate token bucket; no 403 recorded) & Implemented; documented from run logs \\
59 +1{,}342{,}283 sections, 10{,}720{,}890 chunks and embeddings; factor panels for 2{,}501 firms & Implemented, verified \\
60 +1{,}123{,}098 sections submitted to the extractor; ledger free of outage artifacts & Implemented, verified after manual audit of the outage window \\
61 +308{,}065 mentions, 121{,}099 projects, graph of 183{,}472 nodes & Implemented, verified \\
62 +Each mention grounded in its source section & Documented (prompt rule), not mechanically verified \\
63 +Extraction precision/recall, category accuracy, field accuracy, calibration & Not measured (protocol specified) \\
64 +Linkage precision/recall of the resolution key & Not measured; failure modes documented with examples \\
65 +Amounts typed and separated from realized expenditure & Not implemented (proposed) \\
66 +Dated project events and point-in-time revision flags & Not implemented (mention-level timeline only) \\
67 +Human-coded gold standard with two annotators and adjudication & Proposed \\
68 +Author order, affiliations and CRediT contributions & Pending confirmation by the three coauthors \\
69 +\bottomrule
70 +\end{tabularx}
71 +\end{table}
72 +
73 +\section{Additional tables}\label{app:tables}
74 +
75 +\begin{table}[H]
76 +\centering\footnotesize
77 +\caption{Coverage by filing year}\label{tab:coverage_year}
78 +\adjustbox{max width=\textwidth}{\input{tables/tab_coverage_year}}
79 +\end{table}
80 +
81 +\begin{table}[H]
82 +\centering\small
83 +\caption{Issuers with the most resolved projects}\label{tab:top_issuers}
84 +\input{tables/tab_top_issuers}
85 +\end{table}
86 +
87 +\begin{table}[H]
88 +\centering\small
89 +\caption{Self-reported confidence of mentions}\label{tab:confidence}
90 +\input{tables/tab_confidence}
91 +\end{table}
92 +
93 +\begin{table}[H]
94 +\centering\small
95 +\caption{Knowledge graph composition}\label{tab:graph}
96 +\input{tables/tab_graph}
97 +\end{table}
98 +
99 +\section{Replication}\label{app:replication}
100 +
101 +\begin{enumerate}
102 +\item Clone the repository at revision \texttt{2324b58} and create the Python~3.12 environment (\texttt{requirements.txt}; DuckDB~2.0, pyarrow, lxml, sentence-transformers).
103 +\item \texttt{python scripts/build\_universe.py} rebuilds \texttt{data/metadata/universe.json}; the frozen artifact used here is kept with the paper (\texttt{data/}) so that a later IWV file does not change the sample.
104 +\item \texttt{python scripts/download\_filings.py --universe --rate 9} downloads the corpus (idempotent; about 255~GB; seven hours at two IP addresses). The cluster orchestration used for the October 2026 build is in \texttt{scripts/cluster/}.
105 +\item \texttt{python scripts/build\_database.py}, \texttt{parse\_filings.py}, \texttt{build\_embeddings.py}, \texttt{build\_factors.py} build the corpus layer; \texttt{export\_shard.py}/\texttt{merge\_shards.py} are needed only for a distributed build.
106 +\item In \texttt{project-db/}: \texttt{python -m spid pending} reports the sections to be extracted and the estimated cost; \texttt{python run\_full\_extract.py --workers 48} runs the extraction against the ledger; \texttt{python -m spid resolve graph embed analytics} builds the project layer.
107 +\item In the working-paper folder: \texttt{python code/export\_stats.py <repo> data/} and \texttt{python code/make\_tables.py} regenerate every table and figure; \texttt{latexmk} in \texttt{paper/} compiles the manuscript.
108 +\end{enumerate}
109 +
110 +\end{appendices}
added paper/figures/fig_confidence.pdf +0 −0

Binary file not shown.

added paper/figures/fig_confidence.png +0 −0

Binary file not shown.

added paper/figures/fig_coverage.pdf +0 −0

Binary file not shown.

added paper/figures/fig_coverage.png +0 −0

Binary file not shown.

added paper/figures/fig_mentions_year.pdf +0 −0

Binary file not shown.

added paper/figures/fig_mentions_year.png +0 −0

Binary file not shown.

added paper/figures/fig_pipeline.pdf +0 −0

Binary file not shown.

added paper/figures/fig_pipeline.png +0 −0

Binary file not shown.

added paper/figures/fig_types.pdf +0 −0

Binary file not shown.

added paper/figures/fig_types.png +0 −0

Binary file not shown.

added paper/main.bbl +109 −0
@@ -0,0 +1,109 @@
1 +\ifx\undefined\bysame
2 +\newcommand{\bysame}{\leavevmode\hbox to\leftmargin{\hrulefill\,\,}}
3 +\fi
4 +\begin{thebibliography}{xx}
5 +
6 +\harvarditem[Bloom et al.]{Bloom, Hassan, Kalyani, Lerner and
7 + Tahoun}{2021}{bloom2021}
8 +{\bf Bloom, Nicholas, Tarek~A. Hassan, Aakash Kalyani, Josh Lerner, and Ahmed
9 + Tahoun}, ``The Diffusion of Disruptive Technologies,'' NBER Working Paper
10 + 28999, National Bureau of Economic Research 2021.
11 +
12 +\harvarditem[Christen]{Christen}{2012}{christen2012}
13 +{\bf Christen, Peter}, {\it Data Matching: Concepts and Techniques for Record
14 + Linkage, Entity Resolution, and Duplicate Detection}, Berlin: Springer, 2012.
15 +
16 +\harvarditem[Cohen et al.]{Cohen, Malloy and Nguyen}{2020}{cohen2020}
17 +{\bf Cohen, Lauren, Christopher Malloy, and Quoc Nguyen}, ``Lazy Prices,'' {\it
18 + Journal of Finance}, 2020, {\it 75} (3), 1371--1415.
19 +
20 +\harvarditem[Dyer et al.]{Dyer, Lang and Stice-Lawrence}{2017}{dyer2017}
21 +{\bf Dyer, Travis, Mark Lang, and Lorien Stice-Lawrence}, ``The Evolution of
22 + 10-{K} Textual Disclosure: Evidence from {L}atent {D}irichlet {A}llocation,''
23 + {\it Journal of Accounting and Economics}, 2017, {\it 64} (2--3), 221--245.
24 +
25 +\harvarditem[Fellegi and Sunter]{Fellegi and Sunter}{1969}{fellegi1969}
26 +{\bf Fellegi, Ivan~P. and Alan~B. Sunter}, ``A Theory for Record Linkage,''
27 + {\it Journal of the American Statistical Association}, 1969, {\it 64} (328),
28 + 1183--1210.
29 +
30 +\harvarditem[{FTSE Russell}]{{FTSE Russell}}{2026}{ftserussell2026}
31 +{\bf {FTSE Russell}}, ``Russell {US} Equity Indexes: Construction and
32 + Methodology,'' \url{https://www.lseg.com/en/ftse-russell/indices/russell-us}
33 + 2026.
34 +\newblock Accessed October 2026.
35 +
36 +\harvarditem[Gentzkow et al.]{Gentzkow, Kelly and Taddy}{2019}{gentzkow2019}
37 +{\bf Gentzkow, Matthew, Bryan Kelly, and Matt Taddy}, ``Text as Data,'' {\it
38 + Journal of Economic Literature}, 2019, {\it 57} (3), 535--574.
39 +
40 +\harvarditem[Grimmer and Stewart]{Grimmer and Stewart}{2013}{grimmer2013}
41 +{\bf Grimmer, Justin and Brandon~M. Stewart}, ``Text as Data: The Promise and
42 + Pitfalls of Automatic Content Analysis Methods for Political Texts,'' {\it
43 + Political Analysis}, 2013, {\it 21} (3), 267--297.
44 +
45 +\harvarditem[Hassan et al.]{Hassan, Hollander, van Lent and
46 + Tahoun}{2019}{hassan2019}
47 +{\bf Hassan, Tarek~A., Stephan Hollander, Laurence van Lent, and Ahmed Tahoun},
48 + ``Firm-Level Political Risk: Measurement and Effects,'' {\it Quarterly
49 + Journal of Economics}, 2019, {\it 134} (4), 2135--2202.
50 +
51 +\harvarditem[Hoberg and Phillips]{Hoberg and Phillips}{2016}{hoberg2016}
52 +{\bf Hoberg, Gerard and Gordon Phillips}, ``Text-Based Network Industries and
53 + Endogenous Product Differentiation,'' {\it Journal of Political Economy},
54 + 2016, {\it 124} (5), 1423--1465.
55 +
56 +\harvarditem[Hoberg and Maksimovic]{Hoberg and
57 + Maksimovic}{2015}{hobergmaksimovic2015}
58 +{\bf \bysame{} and Vojislav Maksimovic}, ``Redefining Financial Constraints: A
59 + Text-Based Analysis,'' {\it Review of Financial Studies}, 2015, {\it 28} (5),
60 + 1312--1352.
61 +
62 +\harvarditem[Kelly et al.]{Kelly, Papanikolaou, Seru and
63 + Taddy}{2021}{kelly2021}
64 +{\bf Kelly, Bryan, Dimitris Papanikolaou, Amit Seru, and Matt Taddy},
65 + ``Measuring Technological Innovation over the Long Run,'' {\it American
66 + Economic Review: Insights}, 2021, {\it 3} (3), 303--320.
67 +
68 +\harvarditem[Li et al.]{Li, Mai, Shen and Yan}{2021}{li2021}
69 +{\bf Li, Kai, Feng Mai, Rui Shen, and Xinyan Yan}, ``Measuring Corporate
70 + Culture Using Machine Learning,'' {\it Review of Financial Studies}, 2021,
71 + {\it 34} (7), 3265--3315.
72 +
73 +\harvarditem[Loughran and McDonald]{Loughran and McDonald}{2011}{loughran2011}
74 +{\bf Loughran, Tim and Bill McDonald}, ``When Is a Liability Not a Liability?
75 + {T}extual Analysis, Dictionaries, and 10-{K}s,'' {\it Journal of Finance},
76 + 2011, {\it 66} (1), 35--65.
77 +
78 +\harvarditem[Loughran and McDonald]{Loughran and McDonald}{2016}{loughran2016}
79 +{\bf \bysame{} and \bysame{}}, ``Textual Analysis in Accounting and Finance: A
80 + Survey,'' {\it Journal of Accounting Research}, 2016, {\it 54} (4),
81 + 1187--1230.
82 +
83 +\harvarditem[Raasveldt and M{\"u}hleisen]{Raasveldt and
84 + M{\"u}hleisen}{2019}{raasveldt2019}
85 +{\bf Raasveldt, Mark and Hannes M{\"u}hleisen}, ``{DuckDB}: An Embeddable
86 + Analytical Database,'' in ``Proceedings of the 2019 International Conference
87 + on Management of Data (SIGMOD)'' 2019, pp.~1981--1984.
88 +
89 +\harvarditem[Reimers and Gurevych]{Reimers and Gurevych}{2019}{reimers2019}
90 +{\bf Reimers, Nils and Iryna Gurevych}, ``Sentence-{BERT}: Sentence Embeddings
91 + using {S}iamese {BERT}-Networks,'' in ``Proceedings of the 2019 Conference on
92 + Empirical Methods in Natural Language Processing (EMNLP-IJCNLP)'' 2019,
93 + pp.~3982--3992.
94 +
95 +\harvarditem[Tandjambianda et al.]{Tandjambianda, Boucher and
96 + Kammoun}{2026}{spid2026}
97 +{\bf Tandjambianda, Djerrycharli, Simon-Pierre Boucher, and Manel Kammoun},
98 + ``{SPID} --- {SEC} Project Intelligence Database: A Methodological Report on
99 + the {S\&P}~500 Release,'' Internal methodological report, Université du
100 + Québec en Outaouais, Département des sciences administratives 2026.
101 +\newblock Revision df7905d; October 28, 2026.
102 +
103 +\harvarditem[{U.S. Securities and Exchange Commission}]{{U.S. Securities and
104 + Exchange Commission}}{2024}{secfairaccess}
105 +{\bf {U.S. Securities and Exchange Commission}}, ``Accessing {EDGAR} Data: Fair
106 + Access Policy,'' \url{https://www.sec.gov/os/accessing-edgar-data} 2024.
107 +\newblock Accessed October 2026.
108 +
109 +\end{thebibliography}
added paper/main.pdf +0 −0

Binary file not shown.

added paper/main.tex +154 −0
@@ -0,0 +1,154 @@
1 +% ============================================================================
2 +% UQO Working Paper Template — Economics / Finance
3 +% ============================================================================
4 +\documentclass[12pt,letterpaper]{article}
5 +
6 +% --- Encoding & Language ---
7 +\usepackage[utf8]{inputenc}
8 +\usepackage[T1]{fontenc}
9 +\usepackage[english]{babel}
10 +
11 +% --- Page Layout ---
12 +\usepackage[
13 + letterpaper,
14 + top=1in,
15 + bottom=1in,
16 + left=1in,
17 + right=1in
18 +]{geometry}
19 +
20 +% --- Typography ---
21 +\usepackage{mathptmx} % Times-like font (text + math)
22 +\usepackage{microtype} % Improved typographic refinements
23 +\usepackage{setspace}
24 +\onehalfspacing % 1.5 line spacing
25 +
26 +% --- Math ---
27 +\usepackage{amsmath,amssymb,amsthm}
28 +
29 +% --- Tables ---
30 +\usepackage{booktabs}
31 +\usepackage{tabularx}
32 +\usepackage{multirow}
33 +\usepackage{threeparttable}
34 +\usepackage{siunitx}
35 +\sisetup{
36 + input-symbols = {()},
37 + table-format = -1.3,
38 + parse-numbers = false
39 +}
40 +
41 +% --- Figures ---
42 +\usepackage{graphicx}
43 +\usepackage{subcaption}
44 +\usepackage{float}
45 +
46 +% --- Colors & Links ---
47 +\usepackage[dvipsnames]{xcolor}
48 +\usepackage[
49 + colorlinks = true,
50 + linkcolor = NavyBlue,
51 + citecolor = NavyBlue,
52 + urlcolor = NavyBlue,
53 + breaklinks = true
54 +]{hyperref}
55 +
56 +% --- Bibliography ---
57 +\usepackage[authoryear,round,semicolon]{natbib}
58 +\bibliographystyle{aer}
59 +
60 +% --- Captions ---
61 +\usepackage[
62 + font = small,
63 + labelfont = bf,
64 + labelsep = period,
65 + justification = justified,
66 + singlelinecheck = false
67 +]{caption}
68 +
69 +% --- Headers & Footers ---
70 +\usepackage{fancyhdr}
71 +\pagestyle{fancy}
72 +\fancyhf{}
73 +\renewcommand{\headrulewidth}{0pt}
74 +\fancyfoot[C]{\thepage}
75 +
76 +% --- Section Formatting ---
77 +\usepackage{titlesec}
78 +\titleformat{\section}{\large\bfseries}{\thesection.}{0.5em}{}
79 +\titleformat{\subsection}{\normalsize\bfseries}{\thesubsection.}{0.5em}{}
80 +\titleformat{\subsubsection}{\normalsize\itshape}{\thesubsubsection.}{0.5em}{}
81 +
82 +% --- Appendix Support ---
83 +\usepackage[toc,page]{appendix}
84 +
85 +% --- Misc ---
86 +\setlength{\emergencystretch}{3em}
87 +\usepackage{enumitem}
88 +\setlist{nosep,leftmargin=*}
89 +\usepackage{longtable}
90 +\usepackage{tikz}
91 +\usetikzlibrary{positioning,arrows.meta,shapes.misc}
92 +\usepackage{pdflscape}
93 +\usepackage{adjustbox}
94 +
95 +% ============================================================================
96 +% METADATA — Edit these fields for each working paper
97 +% ============================================================================
98 +\newcommand{\WPnumber}{0001}
99 +\newcommand{\WPtitle}{From Corporate Disclosures to a Russell~3000 Strategic Project Database}
100 +\newcommand{\WPsubtitle}{Identification, Measurement, Validation, and Research Applications}
101 +\newcommand{\WPdate}{October 28, 2026}
102 +\newcommand{\WPversion}{1.0 (working draft under validation)}
103 +\newcommand{\WPfunding}{Research funded by the FIRC (Fonds institutionnel de recherche et de création), Université du Québec en Outaouais.}
104 +\newcommand{\WPabstract}{%
105 + We document the construction of SPID, a project-level research database that reconstructs the strategic initiatives disclosed by U.S.-listed companies in their SEC filings. Starting from an issuer universe that merges a current S\&P~500 snapshot with the holdings of the iShares Russell~3000 ETF (2{,}564 distinct SEC registrants), the pipeline collects every 10-K, 10-Q and 8-K filed since January 2010 (527{,}548 documents), segments them into 1.34 million sections, and submits 1.12 million eligible sections to a schema-constrained large-language-model extractor. The result is 308{,}065 evidence-linked project mentions, resolved into 121{,}099 candidate projects for 2{,}492 issuers, each tied to its SEC accession number, section and source passage, together with a controlled 21-category taxonomy, status, cited years, locations, technologies, partners and disclosed amounts. We describe the universe definition and its reconciliation, the distributed acquisition and parsing architecture, the extraction contract and its confidence semantics, the mention-to-project resolution rule and its known failure modes, and the point-in-time structure of the timelines. We then set out a validation framework---stratified human annotation, precision/recall on an independent gold standard, linkage audits and coverage gates---and state explicitly which of its components are implemented and which remain to be executed. Descriptive patterns are reported for the full corpus, with the caveat that a disclosed project is neither a verified investment nor a realized expenditure. The database should be read as a research instrument under validation rather than as an exhaustive census of Russell~3000 projects.%
106 +}
107 +\newcommand{\WPkeywords}{SEC filings, text as data, corporate investment, strategic projects, information extraction, large language models, entity resolution, Russell 3000, measurement validation}
108 +\newcommand{\WPjel}{C55, C81, G31, G38, M41, O33}
109 +
110 +% --- Authors (order, affiliations and contributions to be confirmed by the team; see docs/contributions.md) ---
111 +\newcommand{\WPauthor}{Manel Kammoun \quad Djerrycharli Tandjambianda \quad Simon-Pierre Boucher}
112 +\newcommand{\WPaffiliation}{%
113 + Département des sciences administratives\\
114 + Université du Québec en Outaouais%
115 +}
116 +\newcommand{\WPemail}{%
117 + \href{mailto:manel.kammoun@uqo.ca}{manel.kammoun@uqo.ca} \,\textperiodcentered\,
118 + \href{mailto:djerrycharli.tandjambianda@uqo.ca}{djerrycharli.tandjambianda@uqo.ca} \,\textperiodcentered\,
119 + \href{mailto:simon-pierre.boucher@uqo.ca}{simon-pierre.boucher@uqo.ca}%
120 +}
121 +\newcommand{\WPaddress}{%
122 + Gatineau -- Pavillon Alexandre-Taché\\
123 + 283, boulevard Alexandre-Taché\\
124 + Gatineau, Québec, Canada J9A 1L8%
125 +}
126 +
127 +% ============================================================================
128 +% DOCUMENT
129 +% ============================================================================
130 +\begin{document}
131 +
132 +% --- Title Page ---
133 +\input{sections/titlepage}
134 +
135 +% --- Main Body ---
136 +\input{sections/introduction}
137 +\input{sections/literature}
138 +\input{sections/data}
139 +\input{sections/methodology}
140 +\input{sections/construct}
141 +\input{sections/resolution}
142 +\input{sections/results}
143 +\input{sections/validation}
144 +\input{sections/applications}
145 +\input{sections/conclusion}
146 +
147 +% --- References ---
148 +\newpage
149 +\bibliography{references}
150 +
151 +% --- Appendix ---
152 +\input{appendix/appendix}
153 +
154 +\end{document}
added paper/references.bib +137 −0
Large diff (137 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/applications.tex +25 −0
Large diff (25 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/conclusion.tex +15 −0
Large diff (15 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/construct.tex +35 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/data.tex +54 −0
Large diff (54 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/introduction.tex +15 −0
Large diff (15 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/literature.tex +13 −0
Large diff (13 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/methodology.tex +70 −0
Large diff (70 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/resolution.tex +29 −0
Large diff (29 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/results.tex +107 −0
Large diff (107 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/titlepage.tex +79 −0
Large diff (79 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/validation.tex +50 −0
Large diff (50 lines) is not loaded by default to keep the page fast.Load diff
added paper/tables/prompt.tex +30 −0
Large diff (30 lines) is not loaded by default to keep the page fast.Load diff
added paper/tables/prompt_verbatim.tex +30 −0
Large diff (30 lines) is not loaded by default to keep the page fast.Load diff
added paper/tables/tab_amounts.tex +8 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added paper/tables/tab_confidence.tex +12 −0
Large diff (13 lines) is not loaded by default to keep the page fast.Load diff
added paper/tables/tab_coverage_year.tex +23 −0
Large diff (24 lines) is not loaded by default to keep the page fast.Load diff
added paper/tables/tab_filings.tex +11 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
added paper/tables/tab_graph.tex +14 −0
Large diff (15 lines) is not loaded by default to keep the page fast.Load diff
added paper/tables/tab_items.tex +21 −0
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff
added paper/tables/tab_provenance.tex +14 −0
Large diff (15 lines) is not loaded by default to keep the page fast.Load diff
added paper/tables/tab_reconciliation.tex +19 −0
Large diff (20 lines) is not loaded by default to keep the page fast.Load diff
added paper/tables/tab_sections.tex +11 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
added paper/tables/tab_sectors.tex +17 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added paper/tables/tab_status.tex +11 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
added paper/tables/tab_taxonomy.tex +27 −0
Large diff (27 lines) is not loaded by default to keep the page fast.Load diff
added paper/tables/tab_top_issuers.tex +18 −0
Large diff (19 lines) is not loaded by default to keep the page fast.Load diff
added paper/tables/tab_types.tex +29 −0
Large diff (30 lines) is not loaded by default to keep the page fast.Load diff
added paper/uq_logo.jpg +0 −0

Binary file not shown.

Menu

Theme