Stage E (writing): §1 introduction and §2 literature rewritten to the audited claims and the DOI-verified additions with positioning table (FIRC0001-LITE-01..05, LITE-11, EDIT-10, ADVE-17, THEO-19, ADVE-14, ECON-17); §3 sample with Russell coverage, universe reconciliation and June->October subset (ADVE-09, THEO-07, EDIT-16, AI_C-12, DATA-01, ECON-10, LITE-09, THEO-14, EDIT-06, DATA-09, ADVE-15, EDIT-24); §4 architecture with two parsing paths, 10-K funnel, versioning and ledger facts (EDIT-03, EDIT-04, EDIT-20, ADVE-02, ADVE-07, ADVE-11, AI_C-03, AI_C-07, THEO-06, DATA-02, DATA-13, ECON-03, ECON-08, DATA-08)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Simon-Pierre Boucher committed yesterday (Nov 5, 2026) parent f3b3fa2 Browse files

5 changed files +124 −42

modified paper/sections/data.tex +14 −12

				`@@ -2,20 +2,20 @@`
2	\section{Sample: Issuer Universe and Filing Coverage}\label{sec:sample}	2	\section{Sample: Issuer Universe and Filing Coverage}\label{sec:sample}
3	% ============================================================================	3	% ============================================================================
4		4
5	−\subsection{Estimand: current constituents at a snapshot date}	5	+\subsection{Estimand: current constituents of a proxy at a snapshot date}
6		6
7	−Two different populations can be labeled ``the Russell~3000''. The first is the set of \emph{historical point-in-time constituents}: for each date, the companies that FTSE Russell included in the index after its annual reconstitution \citep{ftserussell2026}. The second is the set of \emph{current constituents at a snapshot date}. The two differ because of index entries and exits, mergers, delistings and IPOs; a panel built on the second population is subject to survivorship and backfilling biases that a point-in-time panel is not. SPID targets the second population. No licensed historical constituent list was available to the team at the time of the build, and FTSE Russell does not publish machine-readable membership lists for free. Any inference about the ``Russell~3000 over 2010--2026'' from this database is therefore an inference about firms that were in the proxy universe in October 2026 and about their filings back to 2010.	7	+Two different populations can be labeled ``the Russell~3000''. The first is the set of \emph{historical point-in-time constituents}: for each date, the companies that FTSE Russell included in the index after its reconstitution \citep{ftserussell2026}. The second is the set of \emph{current constituents at a snapshot date}. The two differ because of index entries and exits, mergers, delistings and IPOs; a panel built on the second population is subject to survivorship and backfilling biases that a point-in-time panel is not. SPID targets the second population, and it observes it through a proxy. No licensed historical constituent list was available to the team at the time of the build, and FTSE Russell does not publish machine-readable membership lists for free; the universe is therefore read from the holdings file of the iShares Russell~3000 ETF (IWV). The index factsheet of \SPIDfactsheetDate\ lists \SPIDrussellIndexConstituents\ constituents \citep{ftserussell3000factsheet2026}; the IWV holdings of \SPIDiwvHoldingsDate\ map to \SPIDrussellDistinctCiks\ SEC registrants, \SPIDrussellProxyCoveragePct\% of that count. The fund's prospectus states that it uses a representative-sampling strategy rather than full replication \citep{iwvholdings2026}, so it need not hold every index member, and which members it omits cannot be determined from public documents; the gap of at most \SPIDrussellGapMax\ securities (\SPIDrussellGapMaxPct\% of the index count) is an upper bound on omitted issuers, and under sampling the omitted names are expected to be the smallest, that is, the extension beyond the S\&P~500 that motivated the build. The three dates differ (index characteristics \SPIDfactsheetDate, holdings \SPIDiwvHoldingsDate, download \SPIDiwvFetchDate), and the index counts securities while the proxy counts issuers after collapsing share classes, so issuer-level coverage is somewhat higher than the printed ratio. Any inference about the ``Russell~3000 over 2010--2026'' from this database is therefore an inference about firms that were in the IWV proxy in October 2026 and about their filings back to 2010.
8		8
9	\subsection{Universe construction and reconciliation}	9	\subsection{Universe construction and reconciliation}
10		10
11	−The issuer universe is frozen in a single JSON artifact, \texttt{data/metadata/universe.json}, built by \texttt{scripts/build\_universe.py} and documented in \texttt{docs/universe\_definition.md}. It merges two sources on the SEC Central Index Key (CIK):	11	+The issuer universe is frozen in a single JSON artefact, \texttt{data/metadata/universe.json}, built by \texttt{scripts/build\_universe.py} and documented in \texttt{docs/universe\_definition.md}; the artefact and its two inputs are archived with the paper with SHA-256 checksums (\texttt{data/metadata/SHA256SUMS}: \texttt{\SPIDuniverseJsonSha\ldots}, \texttt{\SPIDrussellJsonSha\ldots}, \texttt{\SPIDspJsonSha\ldots}). It merges two sources on the SEC Central Index Key (CIK):
12		12
13	\begin{enumerate}	13	\begin{enumerate}
14	−\item the S\&P~500 constituent list scraped from the public Wikipedia table on May~25, 2026 (503 rows, 500 distinct CIKs, with GICS sector and sub-industry), used for the first SPID release; and	14	+\item the S\&P~500 constituent list scraped from the public Wikipedia table on \SPIDspWikipediaDate\ (\SPIDspWikipediaRows\ rows, \SPIDbackfilledCiks\ distinct CIKs, with GICS sector and sub-industry) \citep{wikipediasp500}, used for the first SPID release; and
15	−\item the holdings file of the iShares Russell~3000 ETF (ticker IWV) published by iShares as \texttt{latest-holdings.csv}, with holdings as of October~23, 2026. IWV is an index-tracking fund; its holdings are a close but not exact replica of the index, and the file carries the fund's own sector labels.	15	+\item the holdings file of the iShares Russell~3000 ETF published by iShares as \texttt{latest-holdings.csv}, with holdings as of \SPIDiwvHoldingsDate, downloaded on \SPIDiwvFetchDate\ \citep{iwvholdings2026}. The file carries the fund's own sector labels.
16	\end{enumerate}	16	\end{enumerate}
17		17
18	−Tickers are normalized to the SEC convention (share-class separators become hyphens: ``BRK B'' $\to$ \texttt{BRK-B}) and mapped to CIKs through the SEC's public \texttt{company\_tickers.json}. Lines that are not operating companies (contingent value rights, escrow shares, rights when-issued) and tickers without a CIK (banks that file with the FDIC rather than the SEC) are dropped and listed in the artifact's \texttt{unmatched} field. Table~\ref{tab:reconciliation} gives the full reconciliation from input securities to issuers with resolved projects.	18	+Tickers are normalized to the SEC convention (share-class separators become hyphens: ``BRK B'' $\to$ \texttt{BRK-B}) and mapped to CIKs through the SEC's public \texttt{company\_tickers.json} (\SPIDtickerMapSize\ entries at fetch time). \Cref{tab:reconciliation} gives the reconciliation from input securities to issuers with resolved projects, and two of its steps must be read with care. The first line, \SPIDiwvEquityLines\ equity lines with a ticker, is the count reported by the build log: neither the downloaded holdings file nor the ticker file was archived, the build script drops non-company lines (contingent value rights, escrow shares, rights when-issued) without recording them, and the difference between that count and the \SPIDrussellMappedLines\ mapped lines is therefore not reconstructible line by line. The \SPIDrussellUnmatched\ tickers that reached the mapping step without a CIK are listed in the artefact's \texttt{unmatched} field: four banks that file with the FDIC rather than the SEC and three tickers absent from the SEC ticker file on the download date, for which no reason is recorded. Earlier project documents gave other counts of excluded lines; the artefact count of \SPIDrussellUnmatched\ is the only one that can be verified and is the one used here. The \SPIDrussellMappedLines\ mapped lines collapse to \SPIDrussellDistinctCiks\ CIKs because \SPIDdualClassCiks\ registrants appear with two share classes. The merged universe of \SPIDuniverseN\ issuers is \SPIDbackfilledCiks\ S\&P~500 CIKs plus \SPIDrussellDistinctCiks\ proxy CIKs minus \SPIDuniverseBoth\ present in both lists; \SPIDuniverseSpOnly\ issuers are in the S\&P~500 snapshot but no longer in the IWV file (typically recently acquired) and are retained with an \texttt{in\_sp500} flag.
19		19
20	\begin{table}[H]	20	\begin{table}[H]
21	\centering\small	21	\centering\small
				`@@ -23,16 +23,18 @@ Tickers are normalized to the SEC convention (share-class separators become hyph`
23	\begin{threeparttable}	23	\begin{threeparttable}
24	\input{tables/tab_reconciliation}	24	\input{tables/tab_reconciliation}
25	\begin{tablenotes}\footnotesize	25	\begin{tablenotes}\footnotesize
26	−\item Source: \texttt{data/metadata/universe.json}, \texttt{russell3000\_constituents.json}, \texttt{sp500\_constituents.json} and the merged DuckDB database. The 65 issuers without filings are listed in \texttt{data/issuers\_without\_filings.csv}; most are foreign private issuers (e.g., Spotify, QIAGEN, Canadian Solar, Amdocs) that file annual reports on Form 20-F or 40-F, which are outside the current form scope, plus a few banks whose periodic reports are filed with the FDIC. Seven issuers have filings but no resolved project.	26	+\item Sources: \texttt{data/metadata/universe.json}, \texttt{russell3000\_constituents.json} and \texttt{sp500\_constituents.json} (archived with SHA-256 checksums), the merged DuckDB database, the IWV holdings file of \SPIDiwvHoldingsDate\ \citep{iwvholdings2026} and the Wikipedia S\&P~500 table of \SPIDspWikipediaDate\ \citep{wikipediasp500}. The first line is the count reported by the build log; the holdings file itself was not archived, so the lines dropped before the mapping step are not individually listed. The index held \SPIDrussellIndexConstituents\ constituents on \SPIDfactsheetDate\ \citep{ftserussell3000factsheet2026}; the proxy covers \SPIDrussellProxyCoveragePct\% of that count. The \SPIDissuersWithoutFilings\ issuers without filings are listed in \texttt{data/issuers\_without\_filings.csv}; most are foreign private issuers (e.g., Spotify, QIAGEN, Canadian Solar, Amdocs) that file annual reports on Form 20-F or 40-F, which are outside the current form scope, plus a few banks whose periodic reports are filed with the FDIC. \SPIDissuersFilingsNoProject\ issuers have filings but no resolved project.
27	\end{tablenotes}	27	\end{tablenotes}
28	\end{threeparttable}	28	\end{threeparttable}
29	\end{table}	29	\end{table}
30		30
31	−Several design decisions deserve to be stated plainly. The proxy universe counts \emph{issuers}, not securities: two share classes of the same registrant collapse into one CIK (2{,}587 equity lines map to 2{,}555 CIKs). Companies in the S\&P~500 snapshot but no longer in the IWV file (nine issuers, typically recently acquired) are retained with an \texttt{in\_sp500} flag so that the June 2026 release remains a subset. The universe table records, for each issuer, the SEC name, SIC code and description, state of incorporation, exchange and fiscal-year end from the SEC submissions header, the fund weight, and the cluster node that processed its filings (\texttt{shard\_node}). The brief's recommendation of a \texttt{universe\_snapshots} table with source checksums, membership dates and match confidence is only partially met: snapshot date, source URL and match method are recorded; file hashes and per-row match confidence are not yet stored and are listed as open items in the implementation audit (\texttt{docs/implementation\_audit.md}).	31	+Several design decisions deserve to be stated plainly. The proxy universe counts \emph{issuers}, not securities. The universe table records, for each issuer, the SEC name, SIC code and description, state of incorporation, exchange and fiscal-year end from the SEC submissions header, the fund weight, and the cluster node that processed its filings (\texttt{shard\_node}). The research-design specification against which the build is audited in \texttt{docs/implementation\_audit.md} (hereafter ``the specification'': the list of required tables, keys and validation gates for a research-ready release) calls for a \texttt{universe\_snapshots} table with source checksums, membership dates and match confidence; this is only partially met. Snapshot date, source URL and match method are recorded and the derived artefacts are now hashed, but the source files are not archived and no per-row match confidence is stored, which is why gate~(a) of \Cref{tab:gates} is reported as partially met.
		32	+
		33	+The June 2026 release remains a subset of the October build in the sense that every S\&P~500 issuer is in the universe, but not in the sense of identical counts. The June block yielded \SPIDjuneProjects\ projects from \SPIDbackfilledMentions\ mentions, both reproduced exactly when the resolver is run on the backfilled rows alone; the same \SPIDbackfilledCiks\ issuers now carry \SPIDspProjects\ projects from \SPIDspMentions\ mentions. The increase is explained by scope, not by the resolver: the June run had read only \SPIDjuneTenQFilings\ S\&P~500 10-Qs (\SPIDjuneTenQSections\ sections, \SPIDjuneTenQMentions\ mentions), and the October run added \SPIDoctoberSpTenQSections\ 10-Q sections of \SPIDoctoberSpTenQFilings\ filings for the same issuers (\SPIDoctoberSpTenQMentions\ mentions) plus \SPIDoctoberSpTenKMentions\ 10-K and \SPIDoctoberSpEightKMentions\ 8-K mentions. Of the \SPIDspProjects\ S\&P~500 projects, \SPIDspProjectsJuneOnly\ rest on June mentions only, \SPIDspProjectsBothRuns\ on both runs and \SPIDspProjectsOctoberOnly\ on October mentions only (\Cref{app:runs}).
32		34
33	\subsection{Filing scope}	35	\subsection{Filing scope}
34		36
35	−For every issuer the pipeline enumerates the SEC submissions index (\texttt{data.sec.gov/}\allowbreak\texttt{submissions/}\allowbreak\texttt{CIK\{cik\}.json}, including the paginated archive blocks) and retains the primary document of every Form 10-K, 10-Q and 8-K filed on or after January~1, 2010. Amendments (\texttt{/A}), exhibits other than the primary document, proxy statements, registration statements and foreign-issuer forms are out of scope. The choice reflects where strategic initiatives are described in prose---Items 1, 1A, 2, 7 and 7A of the 10-K, the MD\&A and risk-factor items of the 10-Q, and the body of the 8-K---and it is a defensible but not innocuous restriction: press releases attached as Exhibit~99 to an 8-K, which often carry the first and most detailed announcement of a project, are not read when they are not the primary document. Testing whether this scope omits important announcements is part of the validation roadmap.	37	+For every issuer the pipeline enumerates the SEC submissions index (\texttt{data.sec.gov/}\allowbreak\texttt{submissions/}\allowbreak\texttt{CIK\{cik\}.json}, including the paginated archive blocks) and retains the primary document of every Form 10-K, 10-Q and 8-K filed on or after January~1, 2010. Amendments (\texttt{/A}), exhibits other than the primary document, proxy statements, registration statements and foreign-issuer forms are out of scope. Within the retained documents, the extractor reads Items 1, 1A, 2, 7 and 7A of the 10-K, all seven segmented items of the 10-Q (Part~I Items 1--4, Part~II Items 1, 1A and 2) and the body of the 8-K; 10-K Items 3--6 and 8--9B are parsed but not submitted, and a 10-K in which the parser recognizes no Item heading at all is kept as one \texttt{full\_document} section that is outside the submitted scope (\SPIDfullDocumentTenK\ filings, of which \SPIDfullDocumentTenKLegacy\ were covered by the earlier S\&P~500 ingestion and \SPIDfullDocumentTenKInvisible\ are invisible to SPID; \Cref{sec:architecture}). The choice reflects where strategic initiatives are described in prose, and it is a defensible but not innocuous restriction: press releases attached as Exhibit~99 to an 8-K, which often carry the first and most detailed announcement of a project, are not read when they are not the primary document. Testing whether this scope omits important announcements is part of the validation roadmap.
36		38
37	\begin{table}[H]	39	\begin{table}[H]
38	\centering\small	40	\centering\small
				`@@ -40,15 +42,15 @@ For every issuer the pipeline enumerates the SEC submissions index (\texttt{data`
40	\begin{threeparttable}	42	\begin{threeparttable}
41	\input{tables/tab_filings}	43	\input{tables/tab_filings}
42	\begin{tablenotes}\footnotesize	44	\begin{tablenotes}\footnotesize
43	−\item ``Not parsed'' counts filings whose primary document could not be cleaned (parser error) or was not found on disk after download. Raw HTML size is the sum of primary-document sizes recorded at download time.	45	+\item ``Not parsed'' counts filings whose primary document could not be cleaned (\SPIDparseErrors\ parser errors) or was logged as downloaded but not found on disk (\SPIDfilingsMissingOnDisk\ filings); the \SPIDfilingsNotParsed\ accession numbers are listed in \texttt{results/universe/filings\_status\_missing.csv}. Raw HTML size is the sum of primary-document sizes recorded at download time; no per-file checksum is stored.
44	\end{tablenotes}	46	\end{tablenotes}
45	\end{threeparttable}	47	\end{threeparttable}
46	\end{table}	48	\end{table}
47		49
48	−Table~\ref{tab:filings} summarizes the corpus: 527{,}548 primary documents, 254.5~GB of raw HTML, of which only 16 documents (0.003\%) failed parsing. Figure~\ref{fig:coverage} and Table~\ref{tab:coverage_year} in the appendix show the coverage by year. The number of issuers with at least one filing rises from 1{,}392 in 2010 to 2{,}498 in 2026: the snapshot universe contains many companies that went public after 2010, so the early years of the panel are populated by the older and larger survivors. This is the concrete form taken by the survivorship caveat above.	50	+\Cref{tab:filings} summarizes the corpus: \SPIDfilingsTotal\ primary documents, \SPIDrawGb~GB of raw HTML, of which \SPIDfilingsNotParsed\ documents (\SPIDfilingsNotParsedPct\%) were not parsed. \Cref{fig:coverage} and \Cref{tab:coverage_year} in the appendix show the coverage by year. The number of issuers with at least one filing rises from \SPIDissuersFilingFirstYear\ in 2010 to \SPIDissuersFilingLastYear\ in 2026: the snapshot universe contains many companies that went public after 2010, so the early years of the panel are populated by the older and larger survivors. This is the concrete form taken by the survivorship caveat above.
49		51
50	\begin{figure}[H]	52	\begin{figure}[H]
51	\centering	53	\centering
52	\includegraphics[width=\textwidth]{figures/fig_coverage.pdf}	54	\includegraphics[width=\textwidth]{figures/fig_coverage.pdf}
53	−\caption{Filings and issuers per year in the corpus. The 2026 bar covers January~1 to October~27, 2026. Issuers that went public after 2010 enter the panel at their first filing, so the number of observed issuers increases mechanically over time under the snapshot design.}\label{fig:coverage}	55	+\caption{Filings and issuers per year in the corpus. The 2026 bar covers January~1 to October~27, 2026. Issuers that went public after 2010 enter the panel at their first filing, so the number of observed issuers increases mechanically over time under the snapshot design. Source: \texttt{data/issuers\_by\_year.csv}, \texttt{data/filings\_by\_year\_form.csv}; generated by \texttt{code/make\_tables.py}.}\label{fig:coverage}
54	\end{figure}	56	\end{figure}

modified paper/sections/introduction.tex +6 −6

				`@@ -2,14 +2,14 @@`
2	\section{Introduction}	2	\section{Introduction}
3	% ============================================================================	3	% ============================================================================
4		4
5	−A large share of what firms do strategically---build plants, open data centers, integrate acquisitions, launch products, move systems to the cloud, commit to decarbonization targets---is described, often repeatedly and over several years, in the narrative sections of their regulatory filings. Yet the standard quantitative record of corporate investment is an aggregate: capital expenditure, R\&D expense, acquisitions, or counts of patents and plants. Researchers who want to observe the \emph{unit of decision}---a specific, identifiable initiative with a start, a scope, a budget and a fate---have had to assemble hand-collected samples, rely on commercial databases with opaque coverage, or infer initiatives indirectly from word frequencies. This paper documents an attempt to build that unit systematically, from primary sources, for a broad cross-section of U.S.-listed companies.	5	+A large share of what firms do strategically---build plants, open data centers, integrate acquisitions, launch products, move systems to the cloud, commit to decarbonization targets---is described, often repeatedly and over several years, in the narrative sections of their regulatory filings. The standard quantitative record of corporate investment is nevertheless an aggregate: capital expenditure, R\&D expense, acquisition spending, or counts of patents and plants. Researchers who want to observe the \emph{unit of decision}---a specific, identifiable initiative with a start, a scope, a budget and a fate---have relied on hand-collected samples of announcements, such as the ``Million Dollar Plants'' location decisions of \citet{greenstone2010} or the capital-expenditure announcements studied by \citet{mcconnell1985}, or have inferred activity indirectly from firm-level text scores \citep{hoberg2016,hassan2019}. This paper documents an attempt to build that unit systematically, from primary sources, for a broad cross-section of U.S.-listed companies, and reports what a set of mechanical audits says about the result.
6		6
7	−The object we describe is SPID (\emph{SEC Project Intelligence Database}), developed at the Université du Québec en Outaouais on top of the UQO EDGAR research corpus. Its first release, in June 2026, covered the 500 constituents of the S\&P~500 \citep{spid2026}. The present paper documents its extension, carried out on October~27--28, 2026, to an issuer universe that approximates the Russell~3000: 2{,}564 distinct SEC registrants, 527{,}548 annual, quarterly and current reports filed between January 2010 and October 2026, 1.12 million filing sections submitted to a schema-constrained large-language-model (LLM) extractor, and 308{,}065 resulting project mentions resolved into 121{,}099 candidate projects. Every retained mention carries the SEC accession number, the section identifier and a model-generated description grounded in the source passage, so that any number in the database can be traced back to a public document.	7	+The object we describe is SPID (\emph{SEC Project Intelligence Database}), developed at the Université du Québec en Outaouais on top of the UQO EDGAR research corpus. Its first release, in June 2026, covered the 500 constituents of the S\&P~500 \citep{spid2026}. The present paper documents its extension, carried out on October~27--28, 2026, to an issuer universe that approximates the Russell~3000 through the holdings of an index-tracking fund: \SPIDuniverseN\ distinct SEC registrants, of which \SPIDrussellDistinctCiks\ come from the iShares IWV holdings file and cover \SPIDrussellProxyCoveragePct\% of the \SPIDrussellIndexConstituents\ constituents listed in the index factsheet of \SPIDfactsheetDate\ \citep{ftserussell3000factsheet2026,iwvholdings2026}; \SPIDfilingsTotal\ annual, quarterly and current reports filed between January 2010 and October 2026; \SPIDledgerTotal\ filing sections written to an extraction ledger, of which \SPIDgatePassTotal\ passed a lexical gate and were submitted to a large-language-model (LLM) extractor in JSON-object mode; and \SPIDmentions\ stored project mentions, each linked to its SEC accession number and section identifier, resolved into \SPIDprojectCountCurrentKey\ candidate projects under the current resolution key. The extraction was performed in two runs---June 2026 for the S\&P~500 block and October 2026 for the rest---under one prompt lineage whose text is unchanged by file-system evidence but for which no June code revision exists (\Cref{sec:architecture}, \Cref{app:runs}).
8		8
9	−Our core research question is methodological: can we reliably identify, classify, date, link and quantify distinct strategic projects disclosed by Russell~3000 companies, while preserving source evidence and without claiming exhaustive coverage of real-world activity? The answer offered here is deliberately two-sided. On the positive side, the pipeline is reproducible end to end, every intermediate artifact is versioned, and the full corpus has been processed with a single extraction contract and an audit ledger that records which sections were seen. On the cautious side, several components that a fully validated measurement instrument requires---an independent human-coded gold standard, precision and recall estimates by category and field, an audit of cross-filing linkage---are specified in this paper but have not yet been executed at scale. We therefore describe SPID as a \emph{research database under validation}, and we keep a running register that separates implemented, documented-but-unverified, proposed and empirically verified claims (Appendix~\ref{app:evidence}).	9	+Our research question is methodological: can we reliably identify, classify, date, link and quantify distinct strategic projects disclosed by Russell~3000 companies, while preserving source evidence and without claiming exhaustive coverage of real-world activity? The answer is two-sided and, in this revision, quantified wherever a quantity could be computed without new model calls. On the positive side, the pipeline is re-runnable and ledgered: every submitted section is recorded with its mention count and timestamp, every intermediate artefact is archived with a checksum, and every table, figure and inline number of this paper is regenerated from the exported data by two scripts. A field-level grounding audit over all \SPIDmentions\ mentions finds the extracted years, partners and amounts verbatim in the source section in \SPIDgroundYearsPct\%, \SPIDgroundPartnersPct\% and \SPIDgroundAmountNumberPct\% of cases, and every structured field of a mention in \SPIDgroundAllPct\% of cases (\Cref{sec:validation}). On the cautious side, three results of the same audits bound what the counts mean. First, the focus windows that cap the cost of each call showed the model a median \SPIDmdnaShownMedianPct\% of a submitted 10-K MD\&A, and \SPIDmentionsFromLtTwentyFivePct\% of all mentions come from sections of which less than a quarter was shown; recall on long sections is therefore bounded by construction, not merely unknown. Second, the project count depends on the resolution key: it ranges from \SPIDprojectsKeyTypeDropped\ to \SPIDprojectsKeyStrict\ across the alternative keys of \Cref{sec:resolution}, and \SPIDprojectsKeyedNameTokenPct\% of projects under the current key are grouped on a generic first token of the model's project name. Third, \SPIDfinMentionsPct\% of mentions describe share repurchases, credit facilities, note offerings or similar financing instruments that are not projects in the sense of the construct. The components that a validated measurement instrument requires---precision, recall and category accuracy against independent human labels, and an audit of cross-filing linkage---remain unexecuted: the sampling frame, weights and protocol are delivered with this paper, the labels are not. We therefore continue to describe SPID as a \emph{research database under validation}, and we keep a register that separates implemented, measured, documented-but-unverified and proposed claims (\Cref{app:evidence}).
10		10
11	−The paper makes four contributions. First, it specifies the measurement target precisely: a \emph{project} is a distinct, identifiable, issuer-associated strategic initiative described in one or more disclosures, as opposed to a \emph{project mention} (a particular sourced statement), a \emph{generic strategy} (``we invest in innovation''), a historical or third-party reference, or an uncertain candidate. Section~\ref{sec:construct} develops these distinctions and ties them to the existing 21-category SPID taxonomy, which we document rather than redesign. Second, it documents a scalable architecture: a frozen issuer universe reconciled against SEC identifiers; an acquisition layer that respects the SEC's fair-access limits by sharing one aggregate rate budget across ten cluster nodes and one external gateway; a parser that keeps raw-to-clean-text provenance; and an extraction ledger that makes the paid inference step resumable and idempotent (Section~\ref{sec:architecture}). Third, it exposes the extraction contract and the resolution rule in enough detail to be criticized: the prompt, the JSON schema, the confidence semantics, the deterministic project key, and the ways in which that key can both split and merge real initiatives (Section~\ref{sec:resolution}). Fourth, it lays out a validation protocol and a set of acceptance gates, and reports what the current build does and does not satisfy (Section~\ref{sec:validation}).	11	+The paper makes four contributions. First, it specifies the measurement target: a \emph{project} is a distinct, identifiable, issuer-associated strategic initiative described in one or more disclosures, as opposed to a \emph{project mention} (a particular sourced statement), a \emph{generic strategy} (``we invest in innovation''), a historical or third-party reference, a financing instrument, or an uncertain candidate (\Cref{sec:construct}). Second, it documents an architecture that reads the three main periodic forms for a universe of this size: a frozen issuer universe reconciled against SEC identifiers, an acquisition layer that shares one aggregate rate budget across ten cluster nodes and one external gateway, a parser that keeps section-level provenance, and an extraction ledger that makes the paid inference step resumable (\Cref{sec:architecture}). Third, it exposes the extraction contract and the resolution rule in enough detail to be audited and, in this revision, audits them: the prompt, the JSON output and its deterministic normalisation, the confidence semantics, the project key, and the ways in which that key both splits and merges real initiatives (\Cref{sec:resolution,sec:validation}). Fourth, it lays out a validation protocol with acceptance gates and reports, gate by gate, what the current build satisfies, what the Stage D audits have measured, and what remains an author task (\Cref{sec:validation}). Relative to firm-level text scores \citep{hoberg2016,hassan2019,sautner2023,jha2024} and to event-level extraction from 8-Ks \citep{crane2024,dolphin2026}, SPID's unit is the initiative itself, followed across 10-K, 10-Q and 8-K filings; unlike those studies it does not yet report accuracy against human labels, which \Cref{sec:validation} specifies.
12		12
13	−Three caveats frame everything that follows. A present-day holdings file of an index-tracking fund is a proxy for index membership, not an authoritative historical constituent list; the universe is a snapshot, and survivorship is built into it (Section~\ref{sec:sample}). A disclosed project is evidence that an issuer \emph{described} an initiative, not evidence that the initiative happened, and a disclosed budget is a statement about intentions or estimates, not realized capital expenditure. Finally, the extractor only sees what the parser gives it: sections that were truncated, filings that failed to download, and exhibits outside the selected forms are invisible, and ``zero projects'' for an issuer-year must never be read as ``no projects''.	13	+Three caveats frame everything that follows. A present-day holdings file of an index-tracking fund that uses representative sampling is a proxy for index membership, not an authoritative historical constituent list; the universe is a snapshot, and survivorship is built into it (\Cref{sec:sample}). A disclosed project is evidence that an issuer \emph{described} an initiative, not evidence that the initiative happened, and a disclosed budget is a statement about intentions or estimates, not realized capital expenditure. Finally, the extractor only sees what the parser and the focus windows give it: sections that were truncated, filings that failed to download or parse, exhibits outside the selected forms, and 10-K items outside the five submitted ones are invisible, and ``zero projects'' for an issuer-year must never be read as ``no projects''.
14		14
15	−The rest of the paper is organized as follows. Section~\ref{sec:literature} positions the work in the text-as-data, corporate-investment and information-extraction literatures. Section~\ref{sec:sample} defines the universe and reports its reconciliation. Section~\ref{sec:architecture} describes acquisition, parsing and the distributed build. Section~\ref{sec:construct} defines the project construct and the extraction contract. Section~\ref{sec:resolution} covers mention-to-project resolution, timelines and amounts. Section~\ref{sec:results} reports descriptive patterns. Section~\ref{sec:validation} presents the validation framework and the current audit. Section~\ref{sec:applications} discusses research applications and identification limits, and Section~\ref{sec:conclusion} concludes. Appendices provide the taxonomy, the data dictionary, the prompt, the evidence register and replication instructions.	15	+The rest of the paper is organized as follows. \Cref{sec:literature} positions the work relative to text-based measures of corporate activity, LLM-based measurement and extraction, open EDGAR corpora and record linkage. \Cref{sec:sample} defines the universe and reports its reconciliation and its coverage of the index. \Cref{sec:architecture} describes acquisition, parsing, the distributed build and the two extraction runs. \Cref{sec:construct} defines the project construct, the gate and focus windows, and the extraction contract as implemented. \Cref{sec:resolution} covers mention-to-project resolution, the sensitivity of the project count to the key, timelines and amounts. \Cref{sec:results} reports descriptive patterns. \Cref{sec:validation} presents the audits executed in this revision, the annotation protocol, and the acceptance gates. \Cref{sec:applications} discusses research applications and identification limits, and \Cref{sec:conclusion} lists the limitations, the reproducibility statement and the conclusion. The appendices provide the taxonomy, the prompt and gate, the data-model audit, the evidence register, the inter-revision code diff, additional tables and replication instructions.

modified paper/sections/literature.tex +33 −5

+ \section{Related Literature}\label{sec:literature}
+ % ============================================================================
++\paragraph{Text as data in accounting and finance.} The narrative content of SEC filings has become a standard empirical input. \citet{loughran2011} showed that general-purpose sentiment dictionaries misclassify financial text and introduced finance-specific word lists; \citet{loughran2016} survey the subsequent decade of textual analysis in accounting and finance, and \citet{gentzkow2019} provide the general econometric framework for treating text as data, including the measurement-error and validation issues that any text-derived variable inherits. \citet{dyer2017} document how 10-K disclosure has lengthened and become more repetitive over time, a trend that matters for any extractor that reads the same passages year after year; whether repeated boilerplate produces false-positive projects in SPID has not been measured and is one of the labels of the annotation protocol (\Cref{sec:validation}). \citet{cohen2020} show that changes in otherwise repetitive filing language carry information, which motivates our emphasis on \emph{repeated} mentions of the same initiative across filings rather than on single occurrences. \citet{grimmer2013} offer the general methodological warning that automated text measures must be validated against human judgment on the specific task at hand, which is the stance adopted in \Cref{sec:validation}.
++\paragraph{Measuring corporate activity from disclosures.} Closest to our construct are text-based measures of what firms do rather than how they feel. \citet{hoberg2016} build product-market classifications from 10-K business descriptions; \citet{hobergmaksimovic2015} identify financially constrained firms from the language of the liquidity discussion; \citet{hassan2019} measure firm-level political risk from the share of earnings-call text devoted to politics; \citet{sautner2023} measure firm-level climate exposure from calls and validate it against green hiring and patenting; \citet{bellstam2021} score innovation from analyst reports for firms without patents; \citet{li2021} measure corporate culture with word embeddings; \citet{kelly2021} measure the novelty of patents from their text, and \citet{kalyani2025} track the emergence and diffusion of technologies by discovering bigrams in patents and counting them in earnings calls and job postings. Two recent papers obtain firm-level measures directly from a generative model: \citet{kim2023} summarize earnings-call transcripts into political, climate and AI risk scores, and \citet{jha2024} prompt a chat model to read managers' investment plans from calls, producing a firm-quarter score that is validated against CFO survey responses and predicts capital expenditure several quarters ahead. SPID differs from all of these in its unit of observation. Rather than a firm-level score summarizing the intensity of a theme, it targets \emph{discrete initiatives}---objects with a name, a type, a status, cited years, a location, partners and an amount---that can be followed from filing to filing. Relative to firm-level text scores \citep{hoberg2016,hassan2019,sautner2023,jha2024} and to event-level extraction from 8-Ks \citep{crane2024,dolphin2026}, SPID's unit is the initiative itself, followed across 10-K, 10-Q and 8-K filings; unlike those studies it does not yet report accuracy against human labels, which \Cref{sec:validation} specifies. \Cref{tab:positioning} places the database next to the closest prior work on the dimensions that matter for a user: unit, sources, extractor, provenance granularity and the validation reported.
++\begin{table}[H]
++\centering\scriptsize
++\caption{Positioning of SPID relative to the closest text-based measures, extraction systems and corpora}\label{tab:positioning}
++\begin{threeparttable}
++\begin{tabularx}{\textwidth}{@{}>{\raggedright\arraybackslash}p{2.6cm}>{\raggedright\arraybackslash}p{2.5cm}>{\raggedright\arraybackslash}p{2.1cm}>{\raggedright\arraybackslash}X>{\raggedright\arraybackslash}p{2.2cm}>{\raggedright\arraybackslash}X@{}}
++\toprule
++Study & Unit & Sources & Method & Provenance & Validation reported \\
++\midrule
++\citet{hoberg2016} & firm-pair similarity; industry classification & 10-K business descriptions & word-vector similarity & firm-year & against SIC/NAICS classifications on explaining fundamentals \\
++\citet{hassan2019} & firm-quarter political-risk score & earnings calls & pattern-based bigram share & firm-quarter & against volatility, lobbying and donations; human audit of snippets \\
++\citet{sautner2023} & firm-year climate exposure & earnings calls & machine-learning keyword discovery & firm-year & against green jobs and patents \\
++\citet{kim2023} & firm-level risk measures & earnings calls & GPT summarization & firm-call & against volatility and investment outcomes \\
++\citet{jha2024} & firm-quarter investment score & earnings calls & ChatGPT prompting & firm-quarter & against CFO surveys; predicts capex up to nine quarters ahead \\
++\citet{crane2024} & layoff event & Form 8-K & text extraction & filing & against WARN notices (administrative source) \\
++\citet{dolphin2026} & 8-K event, 119-type three-tier taxonomy & Form 8-K & two-stage LLM, labels traceable to the source text & passage & label reliability evaluated \\
++\citet{loukas2021} & item-segmented corpus & all 10-K filings 1993--2020 & HTML parsing, Item segmentation & item & corpus release, no extraction \\
++\citet{bettencourt2026} & layout-faithful corpus & SEC filings, many forms & reconstruction into structured text with tables preserved & document & corpus release, no extraction \\
++SPID (this paper) & initiative mention, linked into projects across filings & 10-K, 10-Q, 8-K primary documents 2010--2026, \SPIDuniverseN\ issuers & gpt-4o-mini in JSON-object mode behind a lexical gate and focus windows; deterministic key & section (accession, section id); no character span & mechanical grounding of structured fields, 8-K item-code consistency, rules-vs-LLM agreement (\Cref{sec:validation}); no human labels yet \\
++\bottomrule
++\end{tabularx}
++\begin{tablenotes}\footnotesize
++\item Content of the rows other than SPID is taken from the cited works as summarized in the literature review of this revision; it describes the design of each study, not its results. ``Provenance'' is the finest unit at which an output can be traced to its source. The row for SPID describes the build at revision \texttt{2324b58} after the Stage D audits.
++\end{tablenotes}
++\end{threeparttable}
++\end{table}
++\paragraph{Large language models as measurement instruments.} The extraction step relies on an instruction-tuned LLM used in a constrained-output mode: a fixed system prompt, a JSON-object response format whose field list is stated in the prompt rather than enforced by a schema, temperature zero, and an explicit instruction to extract only initiatives grounded in the supplied passage. \citet{ludwig2025} give the econometric framework within which such an output can become a regressor: a measurement generated by a language model is usable for estimation only with a validation subsample that identifies its error structure, and prediction uses require that the model's training data not contain the outcome. SPID currently has the first half of that framework---the instrument and its ledger---and not the second; \Cref{sec:validation} is written as the specification of that validation subsample. \citet{dekok2025} sets out the practical protocol for generative models in accounting research (prompt and model versioning, temperature, repeated runs, validation), several items of which the current build does not meet: no prompt hash or model snapshot is recorded per run and the June model snapshot is unknown (\Cref{sec:architecture}). \citet{gilardi2023} show that zero-shot LLM annotation can match or exceed crowd workers on text-classification tasks, which is the comparison the human-versus-model benchmark of \Cref{sec:validation} is designed to make for this construct; \citet{afzal2026} report accuracy for a generic LLM framework extracting structured fields from narrative financial reports. Closest in design are \citet{dolphin2026}, who label 8-K disclosures against a fine-grained taxonomy with labels constrained to be traceable to the source text and evaluate their reliability, and \citet{crane2024}, who extract one event type from 8-Ks and validate it against an administrative source; both report the reliability that SPID does not yet have. The model is asked to prefer fewer, concrete projects to many vague ones, to return an empty list when no concrete project is described, and to attach a self-reported confidence. We make no claim that this confidence is calibrated: verbalized confidence scores of language models are known to be poorly calibrated \citep{yang2024,ye2024}, a missing confidence is defaulted to a fixed value by the post-processing (\Cref{sec:construct}), and calibration is one of the items of the validation protocol. Semantic search over projects uses sentence embeddings \citep{reimers2019}.
++\paragraph{Open EDGAR corpora.} The corpus layer of SPID re-implements a step that prior work has already published: EDGAR-CORPUS \citep{loukas2021} provides every 10-K from 1993 to 2020 segmented by Item, and the Stanford EDGAR Filings Dataset \citep{bettencourt2026} reconstructs filings into layout-faithful text in which tables are preserved. SPID's parser covers 10-K, 10-Q and 8-K primary documents from 2010 to 2026 for its universe, flattens tables to tab-separated text and stores no character offsets into the raw HTML (\Cref{sec:architecture}); for the S\&P~500 block the 10-K items were prepared by an earlier parser that discarded tables altogether (\Cref{sec:architecture}). A user who needs table-level fidelity should prefer those corpora as the text source and use SPID for its project layer.
++
++\paragraph{Entity resolution and record linkage.} Linking repeated mentions of the same initiative across years is a record-linkage problem in the sense of \citet{fellegi1969}; \citet{christen2012} reviews the modern toolbox of blocking, comparison and classification. The current SPID resolver is a deterministic blocking rule (issuer, project type and a location or name token) rather than a probabilistic linker, and \Cref{sec:resolution} shows how much the project count depends on it. The migration to an explainable, reversible linkage with match scores and human adjudication of ambiguous clusters is part of the roadmap; pre-trained language models have been shown to improve entity matching over string-similarity baselines \citep{li2020}, and generative models can be used for the same task \citep{peeters2023}, which is the direction the proposed \texttt{project\_mention\_links} table is designed to accommodate.
++
++\paragraph{Infrastructure.} The database is implemented as a single analytical file in DuckDB \citep{raasveldt2019}, with Parquet exports; acquisition respects the SEC's published fair-access policy \citep{secfairaccess}, and the Russell~3000 is defined following FTSE Russell's published construction methodology \citep{ftserussell2026}, approximated here through a fund-holdings proxy whose coverage of the index is quantified in \Cref{sec:sample}.

modified paper/sections/methodology.tex +23 −19

				`@@ -4,7 +4,7 @@`
4		4
5	\subsection{Overview}	5	\subsection{Overview}
6		6
7	−Figure~\ref{fig:pipeline} summarizes the pipeline. Two layers are distinguished throughout. The \emph{corpus layer} (code in \texttt{src/} and \texttt{scripts/}) acquires and parses filings, chunks them, embeds the chunks and computes thematic factor scores; it is universe-agnostic and produces a DuckDB database (\texttt{edgar\_research.duckdb}, 68~GB) plus Parquet exports. The \emph{project layer} (code in \texttt{project-db/spid/}) reads the Parquet exports through views, runs candidate gating and LLM extraction, resolves mentions into projects, builds a knowledge graph and project embeddings, and persists everything in a second, much smaller DuckDB file (\texttt{spid.duckdb}, 3.2~GB). The separation matters for reproducibility: the expensive and paid step (LLM extraction) depends only on the frozen Parquet exports and on a ledger of processed sections, so it can be re-run, resumed or audited independently of the corpus build.	7	+\Cref{fig:pipeline} summarizes the pipeline. Two layers are distinguished throughout. The \emph{corpus layer} (code in \texttt{src/} and \texttt{scripts/}) acquires and parses filings, chunks them, embeds the chunks and computes thematic factor scores; it is universe-agnostic and produces a DuckDB database (\texttt{edgar\_research.duckdb}, \SPIDcorpusDbGb~GB) plus Parquet exports (\SPIDcorpusParquetGb~GB of section text). The \emph{project layer} (code in \texttt{project-db/spid/}) reads the Parquet exports through views, runs candidate gating and LLM extraction, resolves mentions into projects, builds a knowledge graph and project embeddings, and persists everything in a second, much smaller DuckDB file (\texttt{spid.duckdb}, \SPIDspidDbGb~GB). The separation matters for re-runs: the paid step (LLM extraction) depends only on the frozen Parquet exports and on a ledger of processed sections, so it can be resumed or audited independently of the corpus build. One exception to the separation, documented in \Cref{sec:segmenters}, is that the 10-K items of the S\&P~500 block were prepared in June 2026 by a parser internal to the project layer and are reused as such.
8		8
9	\begin{figure}[H]	9	\begin{figure}[H]
10	\centering	10	\centering
				`@@ -13,18 +13,18 @@ Figure~\ref{fig:pipeline} summarizes the pipeline. Two layers are distinguished`
13	art/.style={draw, align=center, minimum height=7mm, text width=28mm, fill=yellow!10, font=\scriptsize},	13	art/.style={draw, align=center, minimum height=7mm, text width=28mm, fill=yellow!10, font=\scriptsize},
14	arr/.style={-{Latex[length=2mm]}, thick}]	14	arr/.style={-{Latex[length=2mm]}, thick}]
15	\node[box] (u) {Universe snapshot\\ \scriptsize S\&P~500 $\cup$ IWV $\to$ CIK};	15	\node[box] (u) {Universe snapshot\\ \scriptsize S\&P~500 $\cup$ IWV $\to$ CIK};
16	−\node[box, right=of u] (dl) {SEC acquisition\\ \scriptsize submissions index, primary docs, 10 req/s per IP};	16	+\node[box, right=of u] (dl) {SEC acquisition\\ \scriptsize submissions index, primary docs, \SPIDsecRateLimit\ req/s per IP};
17	\node[box, right=of dl] (p) {Parsing\\ \scriptsize HTML $\to$ text, Item segmentation, chunks};	17	\node[box, right=of dl] (p) {Parsing\\ \scriptsize HTML $\to$ text, Item segmentation, chunks};
18	−\node[box, right=of p] (e) {Embeddings \& factors\\ \scriptsize MiniLM 384-d, 8 anchors};	18	+\node[box, right=of p] (e) {Embeddings \& factors\\ \scriptsize MiniLM \SPIDembeddingDim-d, \SPIDfactorAnchors\ anchors};
19	\node[art, below=of u] (ua) {\texttt{universe.json}\\ \texttt{shards.json}};	19	\node[art, below=of u] (ua) {\texttt{universe.json}\\ \texttt{shards.json}};
20	−\node[art, below=of dl] (da) {\texttt{download\_log.*.jsonl}\\ raw HTML on 10 nodes};	20	+\node[art, below=of dl] (da) {\texttt{download\_log.*.jsonl}\\ raw HTML on \SPIDacqNodes\ nodes};
21	\node[art, below=of p] (pa) {\texttt{filings}, \texttt{filing\_sections}\\ \texttt{filing\_chunks} (Parquet)};	21	\node[art, below=of p] (pa) {\texttt{filings}, \texttt{filing\_sections}\\ \texttt{filing\_chunks} (Parquet)};
22	−\node[art, below=of e] (ea) {\texttt{embeddings/} 16~GB\\ \texttt{chunk\_scores}, panels};	22	+\node[art, below=of e] (ea) {\texttt{embeddings/}\\ \texttt{chunk\_scores}, panels};
23	−\node[box, below=14mm of da, fill=red!5] (x) {Candidate gate + LLM extraction\\ \scriptsize regex gate, focus windows, JSON schema};	23	+\node[box, below=14mm of da, fill=red!5] (x) {Candidate gate + LLM extraction\\ \scriptsize regex gate, focus windows, JSON-object mode};
24	\node[box, right=of x, fill=red!5] (r) {Resolution \& timeline\\ \scriptsize key = CIK $\|$ type $\|$ location/name};	24	\node[box, right=of x, fill=red!5] (r) {Resolution \& timeline\\ \scriptsize key = CIK $\|$ type $\|$ location/name};
25	\node[box, right=of r, fill=red!5] (g) {Graph, project embeddings,\\ analytics, API};	25	\node[box, right=of r, fill=red!5] (g) {Graph, project embeddings,\\ analytics, API};
26	−\node[art, below=of x] (xa) {\texttt{project\_mentions} 308{,}065\\ \texttt{extracted\_sections} 1.12~M};	26	+\node[art, below=of x] (xa) {\texttt{project\_mentions} \SPIDmentions\\ \texttt{extracted\_sections} \SPIDledgerTotal};
27	−\node[art, below=of r] (ra) {\texttt{projects} 121{,}099\\ \texttt{project\_timeline}};	27	+\node[art, below=of r] (ra) {\texttt{projects} \SPIDprojectCountCurrentKey\\ \texttt{project\_timeline}};
28	\node[art, below=of g] (ga) {\texttt{kg\_nodes/edges}\\ \texttt{project\_embeddings}};	28	\node[art, below=of g] (ga) {\texttt{kg\_nodes/edges}\\ \texttt{project\_embeddings}};
29	\draw[arr] (u)--(dl); \draw[arr] (dl)--(p); \draw[arr] (p)--(e);	29	\draw[arr] (u)--(dl); \draw[arr] (dl)--(p); \draw[arr] (p)--(e);
30	\draw[arr] (u)--(ua); \draw[arr] (dl)--(da); \draw[arr] (p)--(pa); \draw[arr] (e)--(ea);	30	\draw[arr] (u)--(ua); \draw[arr] (dl)--(da); \draw[arr] (p)--(pa); \draw[arr] (e)--(ea);
				`@@ -35,36 +35,40 @@ Figure~\ref{fig:pipeline} summarizes the pipeline. Two layers are distinguished`
35	\node[above=1mm of r, yshift=-1mm, font=\scriptsize\itshape] {};	35	\node[above=1mm of r, yshift=-1mm, font=\scriptsize\itshape] {};
36	\node[left=1mm of x, font=\scriptsize\itshape, text width=16mm, align=right] {Project layer (\texttt{spid/})};	36	\node[left=1mm of x, font=\scriptsize\itshape, text width=16mm, align=right] {Project layer (\texttt{spid/})};
37	\end{tikzpicture}	37	\end{tikzpicture}
38	−\caption{Pipeline architecture. Rounded boxes are processing steps; rectangles are persisted artifacts. The project layer reads the Parquet exports of the corpus layer through read-only views and never modifies them.}\label{fig:pipeline}	38	+\caption{Pipeline architecture. Rounded boxes are processing steps; rectangles are persisted artefacts. The project layer reads the Parquet exports of the corpus layer through read-only views and never modifies them; the 10-K items of the S\&P~500 block are the exception described in \Cref{sec:segmenters}. The diagram is drawn by hand (TikZ), not generated from data; the counts shown are the inline macros of \texttt{tables/numbers.tex}.}\label{fig:pipeline}
39	\end{figure}	39	\end{figure}
40		40
41	\subsection{SEC acquisition}	41	\subsection{SEC acquisition}
42		42
43	−The SEC's fair-access policy allows ten requests per second per client and asks for a descriptive \texttt{User-Agent} with a contact address \citep{secfairaccess}. Both constraints shaped the acquisition layer (\texttt{src/acquisition/bulk\_downloader.py}). Requests are issued by a pool of threads behind a single token bucket; the bucket rate is a parameter, and when several processes share one public IP address the orchestrator divides nine requests per second among them. HTTP 403, 429 and 503 responses trigger a 65-second cool-down for the whole process rather than a per-request retry, so that a throttling signal is never answered with a burst. Downloads are idempotent: a filing is skipped when its file exists, and every completed download is appended to a JSON-lines log carrying the accession number, form, filing and report dates, primary document name, size and local path. A trusted-log option lets a node skip filings that an earlier corpus already contains without re-fetching them. The SEC submissions header of each issuer (name, SIC, state of incorporation, exchanges, fiscal-year end, former names) is saved alongside, and feeds the \texttt{companies} table.	43	+The SEC's fair-access policy allows \SPIDsecRateLimit\ requests per second per client and asks for a descriptive \texttt{User-Agent} with a contact address \citep{secfairaccess}. Both constraints shaped the acquisition layer (\texttt{src/acquisition/bulk\_downloader.py}). Requests are issued by a pool of threads behind a single token bucket; the bucket rate is a parameter, and when several processes share one public IP address the orchestrator divides \SPIDacqGatewayDivisor\ requests per second among them. HTTP 403, 429 and 503 responses trigger a \SPIDcooldownSeconds-second cool-down for the whole process rather than a per-request retry, so that a throttling signal is never answered with a burst. Downloads are idempotent: a filing is skipped when its file exists, and every completed download is appended to a JSON-lines log carrying the accession number, form, filing and report dates, primary document name, size and local path. A trusted-log option lets a node skip filings that an earlier corpus already contains without re-fetching them. The SEC submissions header of each issuer (name, SIC, state of incorporation, exchanges, fiscal-year end, former names) is saved alongside, and feeds the \texttt{companies} table.
44		44
45	−The October 2026 build used eleven machines: ten Apple-silicon nodes of the UQO MacLustr cluster behind one residential IP address, and one Linux server in an OVHcloud data center in Beauharnois, Québec, behind a second address. Issuers were assigned to nodes in proportion to CPU cores; half of each node's issuers were downloaded by the node itself (0.9 requests per second each) and half by the external server (one request per second per shard), then shipped to the node over SSH. A first allocation by contiguous CIK ranges proved badly unbalanced---old CIKs are large filers with up to 220 documents each---and was replaced after one hour by an interleaved allocation of the not-yet-started issuers. The whole acquisition (about 400{,}000 new documents) took seven hours at a sustained 17 requests per second, with no fair-access violation recorded.	45	+The October 2026 build used eleven machines: \SPIDacqNodes\ Apple-silicon nodes of the UQO MacLustr cluster behind one residential IP address, and one Linux server in an OVHcloud data center in Beauharnois, Québec, behind a second address. Issuers were assigned to nodes in proportion to CPU cores; half of each node's issuers were downloaded by the node itself (\SPIDacqNodeRate\ requests per second each) and half by the external server (\SPIDacqGatewayRate\ request per second per shard), then shipped to the node over SSH. A first allocation by contiguous CIK ranges proved badly unbalanced---old CIKs are large filers with many documents each---and was replaced after one hour by an interleaved allocation of the not-yet-started issuers. According to the run notes of the research log, the acquisition of \SPIDacqNewDocs\ new documents took \SPIDacqHours\ hours at a sustained \SPIDacqReqPerSec\ requests per second with no fair-access violation recorded; the orchestrator logs behind these figures are not archived with the paper, so they are reported as run facts rather than as exported statistics. Two filings are logged as downloaded but were not found on disk at parse time (\Cref{tab:filings}); in substance they are download failures.
46		46
47	−\subsection{Document processing}	47	+\subsection{Document processing: two parsing paths}\label{sec:segmenters}
48		48
49	−Each primary document is cleaned with \texttt{lxml}: \texttt{script}, \texttt{style}, \texttt{meta} and \texttt{link} elements are removed, tables are flattened to tab-separated text, lists are kept as line items, whitespace is normalized and files above 50~MB are skipped. The segmenter then locates ``Item~\emph{n}'' and ``Part~I/II'' headings by regular expression: 10-Ks are split into up to fourteen sections (Items 1 to 9B), 10-Qs into seven (Part~I Items 1--4, Part~II Items 1, 1A, 2), and 8-Ks are kept as a single \texttt{full\_body} section. A filing in which no heading is found is kept as one \texttt{full\_document} section rather than dropped, so that the foreign-key chain from chunks to sections to filings is never broken. Sections are chunked into 512-word windows with a 64-word overlap for the embedding and factor layers (10.72 million chunks). Provenance is preserved at the section level: each section row stores its item number, order, word and character counts, and each chunk stores its character offsets within the section. Character-level offsets of sections within the raw HTML are \emph{not} stored, a limitation recorded in the audit.	49	+The corpus layer cleans each primary document with \texttt{lxml}: \texttt{script}, \texttt{style}, \texttt{meta} and \texttt{link} elements are removed, tables are flattened to tab-separated text, lists are kept as line items, whitespace is normalized and files above \SPIDmaxFileMb~MB are skipped. The segmenter then locates ``Item~\emph{n}'' and ``Part~I/II'' headings by regular expression: 10-Ks are split into up to \SPIDtenKsectionsMax\ sections (Items 1 to 9B), 10-Qs into \SPIDtenQsectionsN\ (Part~I Items 1--4, Part~II Items 1, 1A, 2), and 8-Ks are kept as a single \texttt{full\_body} section. A filing in which no heading is found is kept as one \texttt{full\_document} section rather than dropped, so that the foreign-key chain from chunks to sections to filings is never broken; for 10-Ks such sections lie outside the submitted scope (\SPIDfullDocumentTenK\ filings), for 10-Qs they are submitted (\SPIDfullDocumentTenQLedgered\ of \SPIDfullDocumentTenQ). Sections are chunked into \SPIDchunkWords-word windows with a \SPIDchunkOverlap-word overlap for the embedding and factor layers (\SPIDchunks\ chunks). Provenance is preserved at the section level: each section row stores its item number, order, word and character counts, and each chunk stores its character offsets within the section. Character-level offsets of sections within the raw HTML are \emph{not} stored, a limitation recorded in the audit.
50		50
51	−Table~\ref{tab:sections} reports the outcome. The 10-K sections are long (median 875 words, mean 5{,}551) and the 10-Q sections somewhat shorter; 8-K bodies are short (median 575 words). Two coverage statistics deserve emphasis because they bound what the extractor could see: 0.003\% of filings failed parsing, and the segmenter's reliance on textual headings means that a 10-K whose items are rendered as images or in unusual markup collapses into a single \texttt{full\_document} section (1{,}230 such 10-K sections were sent to the extractor).	51	+The 10-K items of the S\&P~500 block did not go through this path. They were produced in June 2026 by the project layer's own ingestion module (\texttt{spid/ingest.py}: BeautifulSoup, every \texttt{<table>} element removed, a separate Item regular expression, the longest instance of each Item kept) and stored in a legacy table (\texttt{spid\_sections}, \SPIDlegacyTenKRows\ rows for \SPIDlegacyTenKFilings\ filings); the October project layer keeps the June text for every 10-K already present in that table and reads the corpus-layer sections only for the others. The two segmenters produce measurably different text. On the \SPIDsegPairs\ item pairs available in both tables, the legacy MD\&A has a median of \SPIDlegacyMdnaWords\ words against \SPIDparsedMdnaWords\ for the same item in the corpus layer, \SPIDlegacyMdnaDollar\ dollar signs on average against \SPIDparsedMdnaDollar, and \SPIDlegacyMdnaNumeric\ numeric tokens against \SPIDparsedMdnaNumeric; \SPIDlegacyFewerMoneyPct\% of MD\&A pairs carry fewer money tokens in the legacy text, and the legacy Properties item is less than half the length of its counterpart in \SPIDpropertiesLtHalfPct\% of pairs (\texttt{results/runs/segmenter\_item\_comparison.csv}). Capital-expenditure tables, which the legacy ingestion removed, are among the passages a capex extractor would most want to see. The \SPIDtenKfullBodyRows\ rows labeled \texttt{full\_body} in \Cref{tab:items} are also legacy rows: whole 10-Ks (median \SPIDfullBodyMedianWords\ words) in which the June ingestion found no Item heading, squeezed through the focus windows of \Cref{sec:construct}; the corpus layer has proper items for \SPIDfullBodyWithParsedItems\ of these filings, but those \SPIDfullBodyUnsubmittedSections\ in-scope sections were never submitted because the project layer defers to the legacy table. The earlier version of this paper attributed the \SPIDtenKfullBodyRows\ rows to the October segmenter; that was wrong. The practical consequence is that the S\&P~500 cohort's 10-K text and the Russell-only cohort's 10-K text were prepared by different cleaners, which \Cref{sec:results} and \Cref{app:runs} take into account when the two cohorts are compared.
		52	+
		53	+\Cref{tab:sections} reports the outcome by form. Two coverage statistics bound what the extractor could see: \SPIDfilingsNotParsedPct\% of filings were not parsed, and \SPIDfullDocumentTenKInvisible\ 10-K filings have neither recognized items nor a legacy row and are invisible to the project layer. The 10-K funnel reconciles exactly: \SPIDtenKparsedRows\ parsed 10-K rows minus \SPIDtenKoutOfScopeRows\ out-of-scope items, minus \SPIDtenKlegacyAccessionRows\ rows of filings covered by the legacy table, minus \SPIDtenKunderThirtyRows\ rows under \SPIDminWords\ words, gives \SPIDtenKparsedLedgered\ corpus-layer rows, which with the \SPIDlegacyTenKRows\ legacy rows equal the \SPIDledgerSectionsOneZeroK\ 10-K rows of the ledger.
52		54
53	\begin{table}[H]	55	\begin{table}[H]
54	\centering\small	56	\centering\small
55	−\caption{Parsed sections and sections submitted to the extractor, by form}\label{tab:sections}	57	+\caption{Parsed sections, ledgered sections, gate-passing calls and mentions, by form}\label{tab:sections}
56	\begin{threeparttable}	58	\begin{threeparttable}
57	\adjustbox{max width=\textwidth}{\input{tables/tab_sections}}	59	\adjustbox{max width=\textwidth}{\input{tables/tab_sections}}
58	\begin{tablenotes}\footnotesize	60	\begin{tablenotes}\footnotesize
59	−\item ``Sections sent to the extractor'' counts rows of the \texttt{extracted\_sections} ledger: sections with at least 30 words, restricted for 10-Ks to Items 1, 1A, 2, 7 and 7A (see Table~\ref{tab:items}). Parsed 10-K sections also include Items 3--6, 8, 9--9B, which are not submitted.	61	+\input{tables/tab_sections_note}
60	\end{tablenotes}	62	\end{tablenotes}
61	\end{threeparttable}	63	\end{threeparttable}
62	\end{table}	64	\end{table}
63		65
64	\subsection{Distributed build and merge}	66	\subsection{Distributed build and merge}
65		67
66	−Parsing, chunking and embedding ran on every node against the node's own shard, using the same code revision and a local DuckDB file; each node then exported its tables to Parquet. The master node (M3U96b, which also holds the original S\&P~500 corpus) pulled the shards and merged them with an append-only rule: a row is inserted only if its primary key (accession number, section or chunk UUID) is absent, so that re-running the merge cannot create duplicates. The merged database holds 2{,}564 issuers, 527{,}548 filings, 1{,}342{,}283 sections, 10{,}720{,}890 chunks and as many 384-dimensional embeddings. Parsing throughput was about 100 filings per second per node (five to thirteen minutes per shard); embeddings took 15--40 minutes per node on Apple-silicon GPUs; the merge and the factor computation took seven minutes. The orchestration script (\texttt{scripts/cluster/orchestrate.py}) is stage-based and idempotent: every stage polls marker files on the nodes and can be restarted without side effects. Private infrastructure details (addresses, credentials, keys) are kept outside the repository and outside this paper.	68	+Parsing, chunking and embedding ran on every node against the node's own shard, using the same code revision and a local DuckDB file; each node then exported its tables to Parquet. The master node (M3U96b, which also holds the original S\&P~500 corpus) pulled the shards and merged them with an append-only rule: a row is inserted only if its primary key (accession number, section or chunk UUID) is absent, so that re-running the merge cannot create duplicates. The merged database holds \SPIDuniverseN\ issuers, \SPIDfilingsTotal\ filings, \SPIDsectionsTotal\ sections, \SPIDchunks\ chunks and as many \SPIDembeddingDim-dimensional embeddings. The run notes record a parsing throughput of about \SPIDparseFilingsPerSec\ filings per second per node (\SPIDparseMinutesRange\ minutes per shard), \SPIDembedMinutesRange\ minutes per node for the embeddings on Apple-silicon GPUs, and \SPIDmergeMinutes\ minutes for the merge and the factor computation; like the acquisition figures these are run facts, not exported statistics. The orchestration script (\texttt{scripts/cluster/orchestrate.py}) is stage-based and idempotent: every stage polls marker files on the nodes and can be restarted without side effects. Private infrastructure details (addresses, credentials, keys) are kept outside the repository and outside this paper.
		69	+
		70	+\subsection{Versioning and the two extraction runs}\label{sec:versioning}
67		71
68	−\subsection{Versioning}	72	+The repository is versioned on a private Git server. The revision used for the October 2026 build is \texttt{2324b58} (October~28, 2026). The repository holds \SPIDrepoCommits\ commits, all dated October~27--28, 2026, so no revision of the code exists for the June 2026 run; the earlier version of this paper and the June methodological study \citep{spid2026} cited \texttt{df7905d} as the June revision, but that commit is dated October~27, 2026. What can be established is file-system evidence: the extractor modules (\texttt{llm.py}, \texttt{rules.py}, \texttt{base.py}, \texttt{taxonomy.py}, \texttt{resolve.py}, \texttt{ingest.py}) carry modification times of \SPIDextractorMtimeDate, before the June run, while the driver files (\texttt{config.py}, \texttt{pipeline.py}, \texttt{store.py}) were edited on October~27, 2026 before the first commit, so their June versions are unrecoverable. The diff between the two October revisions on the extractor, parser and driver files is \SPIDdiffStoreLines\ lines in \texttt{store.py} (a DuckDB~2.0 Arrow fallback), and the diff from the first commit adds \SPIDdiffParserLines\ lines in \texttt{filing\_parser.py}; both patches are reproduced in \Cref{app:runs}. The prompt text is therefore unchanged across the two runs as far as the file system can show, which is weaker than a per-run digest.
69		73
70	−The repository is versioned in a private Git server; the revision used for the build described here is \texttt{2324b58} (October~28, 2026), which supersedes revision \texttt{df7905d} examined in the June 2026 methodological study \citep{spid2026}. Model and prompt versions are recorded in the extraction ledger as the backend name; the OpenAI model identifier returned by the API during the October run was \texttt{gpt-4o-mini-2024-07-18}. A hash of the prompt and of the taxonomy is \emph{not} yet stored per run---an item on the audit list---so prompt identity is currently guaranteed by the Git revision rather than by a per-run digest.	74	+The extraction ledger does not help here. Its \texttt{backend} column holds the constant string \texttt{\SPIDledgerBackendValue} for every row of both runs; neither a model identifier nor a prompt, schema or taxonomy hash is stored, and the run logs contain no model string either. The October run is reported by the authors as having used the snapshot \texttt{gpt-4o-mini-2024-07-18}, and the June run the unpinned alias \texttt{gpt-4o-mini}; the first identity rests on the authors' record rather than on an archived artefact, and the June snapshot is unknown. The June block was not re-extracted in October: its \SPIDbackfilledRows\ ledger rows were written at a single timestamp (\SPIDbackfillTimestamp) when the ledger table was created, without the \SPIDminWords-word filter that the October run applied (\SPIDbackfilledUnderThirtyWords\ rows are shorter), which is why the ledger's time range begins on October~27 and why the two runs can be separated only by that timestamp. The proposed \texttt{extraction\_runs} table (\Cref{app:dictionary}) records, per run, the model snapshot, the \texttt{seed} and \texttt{system\_fingerprint} returned by the API, and the prompt and taxonomy hashes; none of these exists for the current build.

modified paper/sections/numbers_pending.tex +48 −0

+ \providecommand{\SPIDsnapshotFilesN}{nine}                    % code/audit/spid_snapshot/SHA256SUMS
+ \providecommand{\SPIDjuneRunDate}{June~18, 2026}              % log_summary.json:june.first/last
+ \providecommand{\SPIDbackfillTimestamp}{October~27, 2026, 13:12}  % backfill_block.json:min_extracted_at
++
++% ---- added for Sections 3-4 (data.tex, methodology.tex) ----
++\providecommand{\SPIDissuersFilingFirstYear}{1{,}392}        % results/timeseries/first_seen_fixed_cohort.csv all_issuers 2010 issuers_filing
++\providecommand{\SPIDissuersFilingLastYear}{2{,}498}         % all_issuers 2026 issuers_filing
++\providecommand{\SPIDfilingsNotParsedPct}{0.003}             % 16 / 527,548
++\providecommand{\SPIDcorpusDbGb}{68}                         % results/*/manifest.json edgar_research.duckdb 73,500,995,584 B (= 68.5 GiB)
++\providecommand{\SPIDspidDbGb}{3.2}                          % spid.duckdb 3,155,701,760 B
++\providecommand{\SPIDcorpusParquetGb}{29.1}                  % results/n0_probe (N0): parquet 1,342,283 sections / 29.1 GB
++\providecommand{\SPIDacqNewDocs}{about 400{,}000}            % author-reported run fact (docs/research_log.md 2026-10-27); orchestrator log not archived with the paper
++\providecommand{\SPIDacqHours}{seven}                        % author-reported run fact; not archived
++\providecommand{\SPIDacqReqPerSec}{17}                       % author-reported run fact; not archived
++\providecommand{\SPIDacqNodes}{ten}                          % docs/research_log.md 2026-10-27 (10 cluster nodes + 1 external gateway)
++\providecommand{\SPIDacqNodeRate}{0.9}                       % author-reported (requests per second per node)
++\providecommand{\SPIDacqGatewayRate}{one}                    % author-reported (requests per second per shard on the gateway)
++\providecommand{\SPIDacqGatewayDivisor}{nine}                % author-reported (requests per second shared behind one IP)
++\providecommand{\SPIDparseFilingsPerSec}{100}                % author-reported run fact; not archived
++\providecommand{\SPIDparseMinutesRange}{five to thirteen}    % author-reported run fact; not archived
++\providecommand{\SPIDembedMinutesRange}{15--40}              % author-reported run fact; not archived
++\providecommand{\SPIDmergeMinutes}{seven}                    % author-reported run fact; not archived
++\providecommand{\SPIDworkersJune}{16}                        % results/ledger/log_summary.json:june.ciks_total_lines (workers=16)
++\providecommand{\SPIDjuneBatchSize}{15}                      % log_summary.json:june (batch=15)
++\providecommand{\SPIDoctoberBatchSize}{20}                   % log_summary.json:october.batch_values
++\providecommand{\SPIDfocusWindowBefore}{220}                 % code/audit/spid_snapshot/llm.py _focus (characters before a gate hit)
++\providecommand{\SPIDfocusWindowAfter}{380}                  % llm.py _focus (characters after a gate hit)
++\providecommand{\SPIDfocusCollectBudget}{12{,}600}           % llm.py _focus (collection stop)
++\providecommand{\SPIDfocusEmitBudget}{9{,}000}               % llm.py _focus (emitted budget)
++\providecommand{\SPIDuserMessageCap}{14{,}000}               % llm.py:126 user message cut
++\providecommand{\SPIDminWords}{30}                           % config.py / pipeline.py minimum section length
++\providecommand{\SPIDconfThreshold}{0.45}                    % llm.py confidence threshold
++\providecommand{\SPIDconfDefault}{0.6}                       % llm.py:142-146 default when confidence is missing or non-numeric
++\providecommand{\SPIDcompletionTokens}{2{,}000}              % llm.py completion limit
++\providecommand{\SPIDlistItemsMax}{eight}                    % llm.py list truncation
++\providecommand{\SPIDlistItemChars}{120}                     % llm.py list item truncation
++\providecommand{\SPIDidNameChars}{40}                        % llm.py:141 anchor = type + name[:40]
++\providecommand{\SPIDretries}{three}                         % llm.py:119-120 retries before returning an empty list
++\providecommand{\SPIDtimeoutSeconds}{120}                    % llm.py call timeout
++\providecommand{\SPIDtenKsectionsMax}{fourteen}              % src/parsing/section_extractor.py (Items 1 to 9B)
++\providecommand{\SPIDtenQsectionsN}{seven}                   % src/parsing/section_extractor.py
++\providecommand{\SPIDchunkWords}{512}                        % src/parsing/chunker.py
++\providecommand{\SPIDchunkOverlap}{64}                       % src/parsing/chunker.py
++\providecommand{\SPIDmaxFileMb}{50}                          % src/parsing/html_parser.py
++\providecommand{\SPIDembeddingDim}{384}                      % MiniLM embedding dimension (scripts/build_embeddings.py)
++\providecommand{\SPIDsecRateLimit}{ten}                      % secfairaccess (requests per second)
++\providecommand{\SPIDcooldownSeconds}{65}                    % src/acquisition/bulk_downloader.py
++\providecommand{\SPIDfactorAnchors}{8}                       % scripts/build_factors.py (thematic anchors)
++\providecommand{\SPIDembeddingsRows}{10{,}720{,}890}         % stats.json chunks
++\providecommand{\SPIDannualPanelRows}{32{,}852}              % data/factor_tables.csv annual_panel rows
++\providecommand{\SPIDquarterlyPanelRows}{128{,}043}          % data/factor_tables.csv quarterly_panel rows

spb/firc0001_uqo

Theme