Stage E (writing): §7 descriptive patterns on stored mentions with financing-class restatement, denominators, cohort/run confound and corrected provenance example (FIRC0001-EDIT-13..15, EDIT-19, ADVE-10, ADVE-13, ADVE-14, AI_C-10, AI_C-15, DATA-06..08, ECON-07, ECON-13..15, LITE-05, LITE-07, LITE-08, THEO-11, THEO-12, THEO-15); §8 validation rewritten around the delivered audits, the annotation frame (no labels) and the gates table (EDIT-01, ADVE-01, AI_C-05, AI_C-14, AI_C-19, THEO-03, ECON-01, ECON-04, THEO-14, ADVE-08); §9 applications and §10 limitations/DAS/conclusion (EDIT-11, AI_C-18, DATA-10, DATA-12, ECON-17); appendix: generated prompt/gate, longtable data model, evidence register, two-runs appendix with patches, notes on every table (EDIT-21, EDIT-22, AI_C-16, AI_C-17, ADVE-16, engine audit)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
7 changed files +253 −101
modified
paper/appendix/appendix.tex
+134 −45
@@ -6,105 +6,194 @@ | |||||
| 6 | 6 | ||||
| 7 | \begin{table}[H] | 7 | \begin{table}[H] | ||
| 8 | \centering\small | 8 | \centering\small | ||
| 9 | −\caption{The 21-category SPID project taxonomy (\texttt{spid/taxonomy.py}, unchanged since the June 2026 release)}\label{tab:taxonomy} | 9 | +\caption{The \SPIDtaxonomyN-category SPID project taxonomy}\label{tab:taxonomy} | ||
| 10 | +\begin{threeparttable} | ||||
| 10 | \input{tables/tab_taxonomy} | 11 | \input{tables/tab_taxonomy} | ||
| 12 | +\begin{tablenotes}\footnotesize | ||||
| 13 | +\item Codes and labels are read with \texttt{ast} from \texttt{spid/taxonomy.py} in the hash-verified snapshot of revision \texttt{2324b58} (\texttt{code/audit/spid\_snapshot/}); the module's modification time predates the June 2026 run and no earlier revision exists, so the taxonomy is the same for both runs as far as the file system can show (\Cref{sec:versioning}). The taxonomy has no category for financing instruments (\Cref{sec:construct_def}); an out-of-vocabulary label returned by the model is mapped by substring matching and, failing that, to \texttt{technology\_deployment}. | ||||
| 14 | +\end{tablenotes} | ||||
| 15 | +\end{threeparttable} | ||||
| 11 | \end{table} | 16 | \end{table} | ||
| 12 | 17 | ||||
| 13 | \section{Extraction contract}\label{app:prompt} | 18 | \section{Extraction contract}\label{app:prompt} | ||
| 14 | 19 | ||||
| 15 | −The system prompt below is passed verbatim to the chat-completion endpoint, with the 21 taxonomy codes and labels substituted at the placeholder. The user message is the string ``\texttt{SECTION: <section\_name>}'' followed by the focused section text, truncated to 14{,}000 characters. Decoding parameters: temperature 0, JSON-object response format, 2{,}000 completion tokens. Model identifier recorded during the October 2026 run: \texttt{gpt-4o-mini-2024-07-18}. | 20 | +The system prompt below is reproduced from \texttt{spid/extract/llm.py} with the \SPIDtaxonomyN\ taxonomy codes and labels substituted exactly as the code does (generated by \texttt{code/make\_tables.py}; never retyped). The user message is the string ``\texttt{SECTION: <section\_name>}'' followed by the focused section text (\Cref{sec:focus}), cut at \SPIDuserMessageCap\ characters. Decoding parameters: temperature 0, JSON-object response format (valid JSON is guaranteed; the field list is not enforced by a schema), \SPIDcompletionTokens\ completion tokens, \SPIDretries\ attempts before an empty list is returned, \SPIDtimeoutSeconds-second timeout, no \texttt{seed}. The model identifier is not recorded in the ledger or the logs (\Cref{sec:versioning}). Straight quotes are rendered as sent. | ||
| 16 | 21 | ||||
| 17 | −\input{tables/prompt} | 22 | +{\footnotesize\input{tables/prompt_verbatim}} | ||
| 18 | 23 | ||||
| 19 | −\noindent The candidate gate is the case-insensitive regular expression \texttt{\textbackslash b(project|initiative|program|invest|expan|construct|build|launch|develop|deploy|implement|acqui|partner|facilit|capacity|capital|migrat|transform|pipeline|plant|data ?cent|automat|sustainab|renewable|restructur)\textbackslash w*}. Sections without a match, or with fewer than 30 words, are not submitted. | 24 | +\noindent The candidate gate is the case-insensitive regular expression below, reproduced from \texttt{spid/extract/rules.py} (\SPIDgateRegexAlternatives\ alternatives, \SPIDgateRegexLength\ characters, compiled with \texttt{\SPIDgateRegexFlags}); line breaks are inserted for display only. Sections without a match, or with fewer than \SPIDminWords\ words, are not submitted; sections with fewer than \SPIDminWords\ words are not ledgered either, except the \SPIDbackfilledUnderThirtyWords\ June rows backfilled without the filter. | ||
| 25 | + | ||||
| 26 | +{\footnotesize\input{tables/gate_regex}} | ||||
| 20 | 27 | ||||
| 21 | \section{Data dictionary and data-model audit}\label{app:dictionary} | 28 | \section{Data dictionary and data-model audit}\label{app:dictionary} | ||
| 22 | 29 | ||||
| 23 | −\begin{table}[H] | 30 | +\Cref{tab:datamodel} lists the tables required by the research-design specification against the current implementation (generated from \texttt{data/datamodel\_audit.csv}). The full field-level dictionary of the current tables is maintained in \texttt{docs/data\_dictionary.md} of the working-paper folder and in the repository documentation; \texttt{amount\_raw} is documented there as a formatted copy of \texttt{amount\_usd}, not as a source string. | ||
| 24 | −\centering\footnotesize | ||||
| 25 | −\caption{Tables required by the research design, current implementation and status}\label{tab:datamodel} | ||||
| 26 | −\begin{tabularx}{\textwidth}{lXl} | ||||
| 27 | −\toprule | ||||
| 28 | −Required table & Current implementation (database, table, key fields) & Status \\ | ||||
| 29 | −\midrule | ||||
| 30 | −\texttt{universe\_snapshots} & \texttt{data/metadata/universe.json} + \texttt{companies} (CIK, ticker(s), name, sector, flags \texttt{in\_sp500}/\texttt{in\_russell3000}, fund weight, SIC, state, exchange, FYE, shard); snapshot date and source URL recorded & Partial: no file hash, no membership dates, no match confidence \\ | ||||
| 31 | −\texttt{issuers} & \texttt{companies} (one row per CIK; former names from SEC header stored in metadata JSON only) & Partial: no name history, no parent/subsidiary links \\ | ||||
| 32 | −\texttt{filings} & \texttt{filings} (accession PK, CIK FK, form, filing/report dates, SEC URL, local path, size, download and parse status) & Implemented; no acceptance timestamp, no checksum \\ | ||||
| 33 | −\texttt{sections} & \texttt{filing\_sections} (UUID PK, accession FK, name, item, order, text, counts) & Implemented; no character offsets into raw HTML \\ | ||||
| 34 | −\texttt{project\_mentions} & \texttt{project\_mentions} (SHA-1 id, accession, CIK, section id, name, type, description, objective, amount, raw amount, lists, years, status, confidence, backend, issuer context) & Implemented; no evidence span, no prompt hash, no five-way label \\ | ||||
| 35 | −\texttt{projects} & \texttt{projects} (MD5 of CIK$|$type$|$location-or-name, representative fields, max amount, counts, first/last seen and cited years) & Implemented (deterministic key) \\ | ||||
| 36 | −\texttt{project\_mention\_links} & implicit in the key (every mention maps to exactly one project) & Proposed: explicit table with score, resolver version, adjudication \\ | ||||
| 37 | −\texttt{project\_events} & \texttt{project\_timeline} (one row per mention: date, form, status, amount, confidence, snippet) & Partial: mention rows, not typed events \\ | ||||
| 38 | −\texttt{project\_amounts} & \texttt{amount\_usd}/\texttt{amount\_raw} on mentions; max on projects & Proposed: type, currency, scale, basis, realization flag \\ | ||||
| 39 | −\texttt{extraction\_runs} & \texttt{extracted\_sections} ledger (section, accession, backend, n\_mentions, timestamp) + run logs & Partial: no prompt/schema/taxonomy hashes, no call status \\ | ||||
| 40 | −\texttt{human\_labels} & --- & Proposed \\ | ||||
| 41 | −\bottomrule | ||||
| 42 | −\end{tabularx} | ||||
| 43 | −\end{table} | ||||
| 44 | 31 | ||||
| 45 | −\noindent The full field-level dictionary of the current tables is maintained in \texttt{docs/data\_dictionary.md} of the working-paper folder and in the repository documentation. | 32 | +{\footnotesize\input{tables/tab_datamodel}} | ||
| 46 | 33 | ||||
| 47 | \section{Evidence register}\label{app:evidence} | 34 | \section{Evidence register}\label{app:evidence} | ||
| 48 | 35 | ||||
| 49 | \begin{table}[H] | 36 | \begin{table}[H] | ||
| 50 | \centering\footnotesize | 37 | \centering\footnotesize | ||
| 51 | −\caption{Evidence register: status of the main claims of this paper at revision \texttt{2324b58}}\label{tab:evidence} | 38 | +\caption{Evidence register: status of the main claims of this paper at revision \texttt{2324b58} after the Stage D audits}\label{tab:evidence} | ||
| 52 | −\begin{tabularx}{\textwidth}{Xl} | 39 | +\begin{threeparttable} | ||
| 40 | +\begin{tabularx}{\textwidth}{@{}>{\raggedright\arraybackslash}X>{\raggedright\arraybackslash}p{5.9cm}@{}} | ||||
| 53 | \toprule | 41 | \toprule | ||
| 54 | Claim & Status \\ | 42 | Claim & Status \\ | ||
| 55 | \midrule | 43 | \midrule | ||
| 56 | −Universe of 2{,}564 issuers built from the IWV holdings file (2026-10-23) and the S\&P~500 cache (2026-05-25) & Implemented, verified (artifact counts) \\ | 44 | +Universe of \SPIDuniverseN\ issuers built from the IWV holdings file (\SPIDiwvHoldingsDate) and the S\&P~500 cache (\SPIDspWikipediaDate); covers \SPIDrussellProxyCoveragePct\% of the index constituent count & Implemented; derived artefacts verified and hashed; source holdings and ticker files not archived; index count read from the factsheet \\ | ||
| 57 | −527{,}548 primary documents of Forms 10-K/10-Q/8-K since 2010 downloaded; 16 unparsed & Implemented, verified (database counts) \\ | 45 | +\SPIDfilingsTotal\ primary documents of Forms 10-K/10-Q/8-K since 2010 downloaded; \SPIDfilingsNotParsed\ not parsed (\SPIDfilingsMissingOnDisk\ missing on disk) & Implemented, verified (database counts; accession list published) \\ | ||
| 58 | −Acquisition respected the SEC rate limit (aggregate token bucket; no 403 recorded) & Implemented; documented from run logs \\ | 46 | +Acquisition respected the SEC rate limit (aggregate token bucket; no 403 recorded) & Implemented; documented from run notes; orchestrator logs not archived \\ | ||
| 59 | −1{,}342{,}283 sections, 10{,}720{,}890 chunks and embeddings; factor panels for 2{,}501 firms & Implemented, verified \\ | 47 | +\SPIDsectionsTotal\ sections, \SPIDchunks\ chunks and embeddings; factor panels for \SPIDannualPanelFirms\ issuers & Implemented, verified (database counts) \\ | ||
| 60 | −1{,}123{,}098 sections submitted to the extractor; ledger free of outage artifacts & Implemented, verified after manual audit of the outage window \\ | 48 | +\SPIDledgerTotal\ sections ledgered; \SPIDgatePassTotal\ calls; outage rows removed reconcile with the log & Implemented, verified (gate recomputed; log reconciled) \\ | ||
| 61 | −308{,}065 mentions, 121{,}099 projects, graph of 183{,}472 nodes & Implemented, verified \\ | 49 | +Ledger free of failed-call artefacts & Not verified: \SPIDllmFailedJsonDecodeError\ silent failures outside the outage; first post-outage batch (\SPIDresumeBatchSections\ sections) suspect; re-submission not authorised \\ | ||
| 62 | −Each mention grounded in its source section & Documented (prompt rule), not mechanically verified \\ | 50 | +Two runs under one prompt text & File-system evidence only (modification times, two-line diff between October revisions); no June revision, no model identifier or prompt hash in the ledger \\ | ||
| 63 | −Extraction precision/recall, category accuracy, field accuracy, calibration & Not measured (protocol specified) \\ | 51 | +\SPIDmentions\ stored mentions (\SPIDledgerMentionsReturned\ returned; \SPIDcollisionMentions\ lost to identifier collisions), \SPIDprojectCountCurrentKey\ projects under the current key, graph of \SPIDkgNodes\ nodes & Implemented, verified; project count depends on the key (\SPIDprojectsKeyTypeDropped--\SPIDprojectsKeyStrict); representative fields depend on tie order \\ | ||
| 64 | −Linkage precision/recall of the resolution key & Not measured; failure modes documented with examples \\ | 52 | +Structured fields of each mention grounded in its source section & Measured for all mentions: \SPIDgroundAllPct\% fully grounded (years \SPIDgroundYearsPct\%, partners \SPIDgroundPartnersPct\%, amounts \SPIDgroundAmountNumberPct\%, locations \SPIDgroundLocationsPct\%, technologies \SPIDgroundTechPct\%); transcription, not correctness \\ | ||
| 65 | −Amounts typed and separated from realized expenditure & Not implemented (proposed) \\ | 53 | +Description grounded in the passage & Documented (prompt rule); not mechanically checkable (no span stored) \\ | ||
| 54 | +What the model was shown & Measured for every ledgered section (median \SPIDmdnaShownMedianPct\% of a 10-K MD\&A; \SPIDmentionsFromLtTwentyFivePct\% of mentions from sections shown $<$ 25\%); recall bound \SPIDrulesFocusRecoveredPct\% \\ | ||||
| 55 | +Category consistent with an external classification & Measured for 8-Ks as a consistency proxy (Item~2.01 $\to$ \texttt{ma\_integration} \SPIDbenchMaGivenTwoOhOne; Item~2.05 $\to$ \texttt{cost\_reduction} \SPIDbenchCostGivenTwoOhFive); not precision or recall \\ | ||||
| 56 | +Agreement with a rule-based baseline & Measured on a weighted sample of \SPIDrulesSampleProductive\ + \SPIDrulesSampleZero\ sections (rules flag \SPIDrulesGivenLlmPct\% of model-productive sections); neither is a ground truth \\ | ||||
| 57 | +Financing instruments and Item~2 swaps as false-positive classes & Measured by pattern (\SPIDfinMentionsPct\% of mentions; \SPIDswapCandidates\ swap candidates); not annotated \\ | ||||
| 58 | +Extraction precision/recall, category accuracy, field accuracy, calibration & Not measured; frame drawn and weighted ($n = \SPIDannNProductive + \SPIDannNZero$), protocol published, labels pending (author task) \\ | ||||
| 59 | +Linkage precision/recall of the resolution key & Not measured; over-merge indicators measured (\Cref{sec:resolution_rule}); audit protocol specified \\ | ||||
| 60 | +Stability under re-submission, prompt paraphrase and model substitution & Not measured; protocol specified; requires model calls not authorised \\ | ||||
| 61 | +Amounts typed and separated from realized expenditure & Not implemented (proposed); \texttt{amount\_raw} is a formatted float; tail audited (\SPIDamountGeFiftyBMentions\ mentions $\geq$ 50 B USD) \\ | ||||
| 66 | Dated project events and point-in-time revision flags & Not implemented (mention-level timeline only) \\ | 62 | Dated project events and point-in-time revision flags & Not implemented (mention-level timeline only) \\ | ||
| 67 | −Human-coded gold standard with two annotators and adjudication & Proposed \\ | 63 | +Release of the derived tables; licence & Author decisions pending \\ | ||
| 68 | Author order, affiliations and CRediT contributions & Pending confirmation by the three coauthors \\ | 64 | Author order, affiliations and CRediT contributions & Pending confirmation by the three coauthors \\ | ||
| 69 | \bottomrule | 65 | \bottomrule | ||
| 70 | \end{tabularx} | 66 | \end{tabularx} | ||
| 67 | +\begin{tablenotes}\footnotesize | ||||
| 68 | +\item ``Measured'' entries refer to the read-only audits of \texttt{code/audit/} whose outputs are in \texttt{results/} with a manifest (script hash, database sizes and modification times, row counts); none involved a model call. ``Verified'' means recomputed from the exported data or the databases; ``documented'' means stated by the code or the run notes without an independent check. | ||||
| 69 | +\end{tablenotes} | ||||
| 70 | +\end{threeparttable} | ||||
| 71 | +\end{table} | ||||
| 72 | + | ||||
| 73 | +\section{The two extraction runs}\label{app:runs} | ||||
| 74 | + | ||||
| 75 | +\paragraph{Git and file-system evidence.} The repository holds \SPIDrepoCommits\ commits, all dated October~27--28, 2026 (\texttt{results/runs/git\_evidence.md}); \texttt{project-db/} enters at the first commit. The extractor modules carry modification times of \SPIDextractorMtimeDate\ and SHA-256 hashes recorded in \texttt{code/audit/spid\_snapshot/SHA256SUMS}; the driver files were edited on October~27, 2026 before the first commit. The diff between the revision the earlier version called ``the June revision'' (\texttt{df7905d}, dated October~27) and the build revision \texttt{2324b58} on the extractor, parser and driver files is the following change to \texttt{store.py}: | ||||
| 76 | + | ||||
| 77 | +{\scriptsize\verbatiminput{../results/runs/git_diff_df7905d_2324b58.patch}} | ||||
| 78 | + | ||||
| 79 | +\noindent The diff from the first commit (\texttt{0304133}) to \texttt{2324b58} on the same files is the following change to the corpus-layer \texttt{filing\_parser.py}: | ||||
| 80 | + | ||||
| 81 | +{\scriptsize\verbatiminput{../results/runs/git_diff_0304133_2324b58.patch}} | ||||
| 82 | + | ||||
| 83 | +\paragraph{Mentions and projects by cohort, run and form.} \Cref{tab:june_october} reconciles the mentions of the S\&P~500 cohort (\texttt{in\_sp500} true) and of the Russell-only cohort by run and form; the June block is identified by the backfill timestamp. The S\&P~500 cohort's 10-Q mentions come from both runs for the same issuers, which permits a within-issuer comparison: on the \SPIDpairedIssuers\ issuers with 10-Qs in both runs, the productive share is \SPIDwithinProductiveJune\% in June and \SPIDwithinProductiveOctober\% in October (paired difference \SPIDpairedDeltaPp\ percentage points, standard error \SPIDpairedSePp), mentions per row \SPIDwithinMentionsPerRowJune\ and \SPIDwithinMentionsPerRowOctober, mean confidence \SPIDwithinConfJune\ and \SPIDwithinConfOctober, and the M\&A share \SPIDwithinMaShareJune\% and \SPIDwithinMaShareOctober\% (\texttt{results/runs/within\_sp500\_*.csv}). For 10-Qs, whose text went through the same parser in both runs, the two runs therefore behave alike; the cohort difference of \Cref{sec:time} comes from the volume of sections and from the 10-K segmenter, not from the extractor. | ||||
| 84 | + | ||||
| 85 | +\begin{table}[H] | ||||
| 86 | +\centering\scriptsize | ||||
| 87 | +\caption{Mentions, projects touched, filings and issuers by cohort, extraction run and form}\label{tab:june_october} | ||||
| 88 | +\begin{threeparttable} | ||||
| 89 | +\adjustbox{max width=\textwidth}{\input{tables/tab_june_october}} | ||||
| 90 | +\begin{tablenotes}\footnotesize | ||||
| 91 | +\item Source: \texttt{results/resolver/june\_october\_reconciliation.csv} (\texttt{code/audit/a03\_resolver\_speccurve.py}). \texttt{in\_sp500} is the cohort flag of the universe; \texttt{run} is ``backfilled\_june'' for rows stamped at the backfill timestamp and ``october'' otherwise; ``projects touched'' counts the projects of the current key that have at least one mention in the cell, so a project can be touched by both runs. The issuer-level regression of the project count on the cohort flag is in \texttt{results/runs/table6\_cohort\_regression.json} and summarized in \Cref{sec:time}. | ||||
| 92 | +\end{tablenotes} | ||||
| 93 | +\end{threeparttable} | ||||
| 71 | \end{table} | 94 | \end{table} | ||
| 72 | 95 | ||||
| 96 | +\paragraph{Fiscal year.} \texttt{fiscal\_year} is empty for \SPIDfiscalYearNullMentions\ mentions because the corresponding \SPIDfiscalYearNullFilings\ filings carry no fiscal year in the corpus layer; a derived fill from the report or filing date is exported for \SPIDfiscalYearFillAccessions\ accessions (\texttt{results/runs/fiscal\_year\_fill\_by\_accession.csv}) and is not stored in the database. | ||||
| 97 | + | ||||
| 73 | \section{Additional tables}\label{app:tables} | 98 | \section{Additional tables}\label{app:tables} | ||
| 74 | 99 | ||||
| 75 | \begin{table}[H] | 100 | \begin{table}[H] | ||
| 76 | \centering\footnotesize | 101 | \centering\footnotesize | ||
| 77 | \caption{Coverage by filing year}\label{tab:coverage_year} | 102 | \caption{Coverage by filing year}\label{tab:coverage_year} | ||
| 103 | +\begin{threeparttable} | ||||
| 78 | \adjustbox{max width=\textwidth}{\input{tables/tab_coverage_year}} | 104 | \adjustbox{max width=\textwidth}{\input{tables/tab_coverage_year}} | ||
| 105 | +\begin{tablenotes}\footnotesize | ||||
| 106 | +\item Source: \texttt{data/issuers\_by\_year.csv}, \texttt{data/filings\_by\_year\_form.csv}, \texttt{data/mentions\_by\_year.csv}, \texttt{data/projects\_by\_first\_seen\_year.csv}. ``Issuers'' counts issuers with at least one filing in the year; ``Projects first seen'' counts projects of the current key whose earliest mention is filed in the year, so 2010 is left-censored and 2026 (to \SPIDfilingDateLast) is partial. The fixed-cohort and windowed-key series are in \texttt{results/timeseries/first\_seen\_fixed\_cohort.csv} and \Cref{tab:timeseries_cohort}. | ||||
| 107 | +\end{tablenotes} | ||||
| 108 | +\end{threeparttable} | ||||
| 109 | +\end{table} | ||||
| 110 | + | ||||
| 111 | +\begin{table}[H] | ||||
| 112 | +\centering\scriptsize | ||||
| 113 | +\caption{First-seen projects and mentions per issuer, all issuers and the fixed 2010 cohort, under the current key and under windowed keys}\label{tab:timeseries_cohort} | ||||
| 114 | +\begin{threeparttable} | ||||
| 115 | +\adjustbox{max width=\textwidth, max totalheight=0.82\textheight}{\input{tables/tab_timeseries_cohort}} | ||||
| 116 | +\begin{tablenotes}\footnotesize | ||||
| 117 | +\item Source: \texttt{results/timeseries/first\_seen\_fixed\_cohort.csv} (\texttt{code/audit/a09\_timeseries\_cohort.py}). ``fixed\_2010\_cohort'' restricts every series to the \SPIDfixedCohortIssuers\ issuers that file in every year from 2010 to 2025. ``first\_seen\_k3'' and ``first\_seen\_k5'' open a new episode when a project key is silent for more than three or five years (\SPIDepisodesKthree\ and \SPIDepisodesKfive\ episodes in total against \SPIDprojectCountCurrentKey\ projects). 2010 is left-censored and 2026 partial. | ||||
| 118 | +\end{tablenotes} | ||||
| 119 | +\end{threeparttable} | ||||
| 79 | \end{table} | 120 | \end{table} | ||
| 80 | 121 | ||||
| 81 | \begin{table}[H] | 122 | \begin{table}[H] | ||
| 82 | \centering\small | 123 | \centering\small | ||
| 83 | \caption{Issuers with the most resolved projects}\label{tab:top_issuers} | 124 | \caption{Issuers with the most resolved projects}\label{tab:top_issuers} | ||
| 125 | +\begin{threeparttable} | ||||
| 84 | \input{tables/tab_top_issuers} | 126 | \input{tables/tab_top_issuers} | ||
| 127 | +\begin{tablenotes}\footnotesize | ||||
| 128 | +\item Source: \texttt{data/top\_issuers.csv}. Counts are projects of the current key; rows with equal counts are ordered by ticker. The list is consistent with issuers whose business consists of enumerable sites, routes or wells; whether it also reflects over-splitting by the key has not been measured (\Cref{sec:resolution_rule}). | ||||
| 129 | +\end{tablenotes} | ||||
| 130 | +\end{threeparttable} | ||||
| 85 | \end{table} | 131 | \end{table} | ||
| 86 | 132 | ||||
| 87 | \begin{table}[H] | 133 | \begin{table}[H] | ||
| 88 | \centering\small | 134 | \centering\small | ||
| 89 | \caption{Self-reported confidence of mentions}\label{tab:confidence} | 135 | \caption{Self-reported confidence of mentions}\label{tab:confidence} | ||
| 136 | +\begin{threeparttable} | ||||
| 90 | \input{tables/tab_confidence} | 137 | \input{tables/tab_confidence} | ||
| 138 | +\begin{tablenotes}\footnotesize | ||||
| 139 | +\item Source: \texttt{data/confidence\_dist.csv}; confidence rounded to one decimal over the \SPIDmentions\ stored mentions. Mentions below \SPIDconfThreshold\ are not stored. The 0.6 bin includes \SPIDconfExactDefault\ mentions whose confidence was defaulted by the post-processing because the model returned none (\Cref{sec:contract}); the minimum stored value in both runs is 0.5. No calibration has been measured. | ||||
| 140 | +\end{tablenotes} | ||||
| 141 | +\end{threeparttable} | ||||
| 91 | \end{table} | 142 | \end{table} | ||
| 92 | 143 | ||||
| 93 | \begin{table}[H] | 144 | \begin{table}[H] | ||
| 94 | \centering\small | 145 | \centering\small | ||
| 95 | \caption{Knowledge graph composition}\label{tab:graph} | 146 | \caption{Knowledge graph composition}\label{tab:graph} | ||
| 147 | +\begin{threeparttable} | ||||
| 96 | \input{tables/tab_graph} | 148 | \input{tables/tab_graph} | ||
| 149 | +\begin{tablenotes}\footnotesize | ||||
| 150 | +\item Source: \texttt{data/kg.csv}. Nodes are projected from the \texttt{projects} table of the current key (\texttt{owns}: company to project; \texttt{located\_in}: project to canonical location; \texttt{uses}: project to technology tag). Partner nodes are created without edges. Technology tags are free text and up to \SPIDtechRegexEqualPct\% of tag lists may have been back-filled from the model's description (\Cref{sec:contract}). | ||||
| 151 | +\end{tablenotes} | ||||
| 152 | +\end{threeparttable} | ||||
| 153 | +\end{table} | ||||
| 154 | + | ||||
| 155 | +\begin{table}[H] | ||||
| 156 | +\centering\scriptsize | ||||
| 157 | +\caption{Most frequent name tokens of the resolution key among location-less mentions}\label{tab:top_name_tokens} | ||||
| 158 | +\begin{threeparttable} | ||||
| 159 | +\input{tables/tab_top_name_tokens} | ||||
| 160 | +\begin{tablenotes}\footnotesize | ||||
| 161 | +\item Source: \texttt{results/resolver/top\_name\_tokens.csv} (\texttt{code/audit/a03\_resolver\_speccurve.py}), first twenty tokens. The token is the first word of at least four letters in the normalized project name of a mention without an extracted location (\Cref{sec:resolution_rule}); ``pct of token-keyed'' shares are over the \SPIDmentionsKeyedNameToken\ mentions and \SPIDprojectsKeyedNameToken\ projects keyed on a token. | ||||
| 162 | +\end{tablenotes} | ||||
| 163 | +\end{threeparttable} | ||||
| 164 | +\end{table} | ||||
| 165 | + | ||||
| 166 | +\begin{table}[H] | ||||
| 167 | +\centering\scriptsize | ||||
| 168 | +\caption{Over-merge indicators of the resolution key, by category}\label{tab:overmerge} | ||||
| 169 | +\begin{threeparttable} | ||||
| 170 | +\adjustbox{max width=\textwidth}{\input{tables/tab_overmerge}} | ||||
| 171 | +\begin{tablenotes}\footnotesize | ||||
| 172 | +\item Source: \texttt{results/resolver/overmerge\_by\_type.csv}. ``pct\_ge3\_names'' (``pct\_ge2\_names'') is the share of projects of the category whose mentions carry three (two) or more distinct project names; ``pct\_ge3\_partners\_among\_with\_partner'' is the share, among projects with at least one partner, whose mentions name three or more distinct first partners; ``pct\_multi\_all\_names\_distinct'' is the share of multi-mention projects in which every mention carries a different name. Three or more distinct partners in one M\&A project is an over-merge by construction; distinct names may be renamings of one initiative or distinct initiatives, which only annotation can separate. | ||||
| 173 | +\end{tablenotes} | ||||
| 174 | +\end{threeparttable} | ||||
| 175 | +\end{table} | ||||
| 176 | + | ||||
| 177 | +\begin{table}[H] | ||||
| 178 | +\centering\scriptsize | ||||
| 179 | +\caption{Field-level grounding of the structured fields in the source section, by form, item and field}\label{tab:grounding} | ||||
| 180 | +\begin{threeparttable} | ||||
| 181 | +\adjustbox{max width=\textwidth, max totalheight=0.84\textheight}{\input{tables/tab_grounding}} | ||||
| 182 | +\begin{tablenotes}\footnotesize | ||||
| 183 | +\item Source: \texttt{results/grounding/grounding\_by\_field\_form\_item.csv} (\texttt{code/audit/a02\_grounding.py}, all \SPIDmentions\ mentions joined to their stored section text; \SPIDgroundBootstrapReps\ bootstrap replicates clustered by section). A mention is ``grounded'' on a field when every element of the field is found in the section text after case folding (years as four-digit strings; locations, partners and technologies as literal strings; amounts in the usual renderings, e.g.\ ``1.5 billion'', ``1,500 million'', ``\$1.5B'', with one to three significant digits). Grounding measures transcription, not the correctness of the mention; the lower rates for locations and technologies reflect normalization by the model as well as invention; and for years and technologies the stored list may have been back-filled from the model's description (\Cref{sec:contract}). | ||||
| 184 | +\end{tablenotes} | ||||
| 185 | +\end{threeparttable} | ||||
| 97 | \end{table} | 186 | \end{table} | ||
| 98 | 187 | ||||
| 99 | \section{Replication}\label{app:replication} | 188 | \section{Replication}\label{app:replication} | ||
| 100 | 189 | ||||
| 101 | \begin{enumerate} | 190 | \begin{enumerate} | ||
| 102 | −\item Clone the repository at revision \texttt{2324b58} and create the Python~3.12 environment (\texttt{requirements.txt}; DuckDB~2.0, pyarrow, lxml, sentence-transformers). | 191 | +\item Clone the repository at revision \texttt{2324b58} and create the Python environment from the lock files shipped with the paper (\texttt{.python-version} = \SPIDpythonVersion; \texttt{requirements.lock}, \SPIDrequirementsLockPackages\ packages from the master node, including DuckDB \SPIDduckdbVersion, pyarrow, lxml and sentence-transformers; \texttt{environment.md} records the hardware). The exhibit environment of the paper is pinned separately in \texttt{reproducibility/environment/}. | ||
| 103 | −\item \texttt{python scripts/build\_universe.py} rebuilds \texttt{data/metadata/universe.json}; the frozen artifact used here is kept with the paper (\texttt{data/}) so that a later IWV file does not change the sample. | 192 | +\item \texttt{python scripts/build\_universe.py} rebuilds \texttt{data/metadata/universe.json} from a \emph{current} download of the holdings and ticker files, which will not reproduce the October 2026 snapshot; the frozen artefacts used here are kept with the paper (\texttt{data/metadata/}, SHA-256 in \texttt{SHA256SUMS}) and must be used instead. | ||
| 104 | −\item \texttt{python scripts/download\_filings.py --universe --rate 9} downloads the corpus (idempotent; about 255~GB; seven hours at two IP addresses). The cluster orchestration used for the October 2026 build is in \texttt{scripts/cluster/}. | 193 | +\item \texttt{python scripts/download\_filings.py --universe --rate 9} downloads the corpus (idempotent; about \SPIDrawGb~GB; \SPIDacqHours\ hours at two IP addresses in October 2026). The cluster orchestration used for the October 2026 build is in \texttt{scripts/cluster/}. | ||
| 105 | −\item \texttt{python scripts/build\_database.py}, \texttt{parse\_filings.py}, \texttt{build\_embeddings.py}, \texttt{build\_factors.py} build the corpus layer; \texttt{export\_shard.py}/\texttt{merge\_shards.py} are needed only for a distributed build. | 194 | +\item \texttt{python scripts/build\_database.py}, \texttt{parse\_filings.py}, \texttt{build\_embeddings.py}, \texttt{build\_factors.py} build the corpus layer; \texttt{export\_shard.py}/\texttt{merge\_shards.py} are needed only for a distributed build. The S\&P~500 10-K items of the June 2026 block cannot be rebuilt by this path (\Cref{sec:segmenters}). | ||
| 106 | −\item In \texttt{project-db/}: \texttt{python -m spid pending} reports the sections to be extracted and the estimated cost; \texttt{python run\_full\_extract.py --workers 48} runs the extraction against the ledger; \texttt{python -m spid resolve graph embed analytics} builds the project layer. | 195 | +\item In \texttt{project-db/}: \texttt{python -m spid pending} reports the sections to be extracted and the estimated cost; \texttt{python run\_full\_extract.py --workers \SPIDworkers} runs the extraction against the ledger (a re-run is not expected to return identical mentions; \Cref{sec:contract}); \texttt{python -m spid resolve graph embed analytics} builds the project layer (representative fields depend on tie order; \Cref{sec:resolution_rule}). | ||
| 107 | −\item In the working-paper folder: \texttt{python code/export\_stats.py <repo> data/} and \texttt{python code/make\_tables.py} regenerate every table and figure; \texttt{latexmk} in \texttt{paper/} compiles the manuscript. | 196 | +\item In the working-paper folder: \texttt{./run\_all.sh} regenerates every table, figure and inline number from \texttt{data/} and \texttt{results/} and runs the \SPIDtestsN\ checks of \texttt{tests/}; \texttt{./run\_all.sh --with-node} first re-runs \texttt{code/export\_stats.py} read-only on the master node, and \texttt{--with-audit} re-runs the Stage D audits of \texttt{code/audit/} (read-only connections, temporary tables only, seed \SPIDannSeed). \texttt{latexmk} in \texttt{paper/} compiles the manuscript. | ||
| 108 | \end{enumerate} | 197 | \end{enumerate} | ||
| 109 | 198 | ||||
| 110 | \end{appendices} | 199 | \end{appendices} | ||
modified
paper/main.tex
+1 −0
@@ -93,6 +93,7 @@ | |||||
| 93 | \usepackage{pdflscape} | 93 | \usepackage{pdflscape} | ||
| 94 | \usepackage{adjustbox} | 94 | \usepackage{adjustbox} | ||
| 95 | \usepackage{upquote} % straight quotes in verbatim (extraction contract, gate regex) | 95 | \usepackage{upquote} % straight quotes in verbatim (extraction contract, gate regex) | ||
| 96 | +\usepackage{verbatim} % \verbatiminput of the inter-revision patches (Appendix, results/runs/*.patch) | ||||
| 96 | \input{tables/numbers} % generated inline-number macros (code/make_tables.py); never edit by hand | 97 | \input{tables/numbers} % generated inline-number macros (code/make_tables.py); never edit by hand | ||
| 97 | \input{sections/numbers_pending} % PROVISIONAL \providecommand macros for Stage D audit numbers not yet emitted by the generator (Agent 10); each line names its results/ source | 98 | \input{sections/numbers_pending} % PROVISIONAL \providecommand macros for Stage D audit numbers not yet emitted by the generator (Agent 10); each line names its results/ source | ||
| 98 | 99 | ||||
modified
paper/sections/applications.tex
+9 −8
@@ -4,22 +4,23 @@ | |||||
| 4 | 4 | ||||
| 5 | \subsection{What the database can support} | 5 | \subsection{What the database can support} | ||
| 6 | 6 | ||||
| 7 | −Four families of empirical designs are natural once the validation gates are passed; each is listed with the restriction it must respect. | 7 | +Four families of empirical designs are natural once the validation gates are passed; each is listed with the restriction it must respect now. | ||
| 8 | 8 | ||||
| 9 | −\paragraph{Project counts and composition as firm-year variables.} The issuer-year count of newly observed projects, by category, is a text-derived measure of disclosed strategic activity comparable in spirit to the thematic scores of the factor layer but with a discrete, interpretable unit. Because the panel is a snapshot universe, firm fixed effects and within-firm comparisons are safer than cross-sectional levels, and the later entry of smaller issuers (Figure~\ref{fig:coverage}) should be modeled explicitly. | 9 | +\paragraph{Project counts and composition as firm-year variables.} The issuer-year count of newly observed projects, by category, is a text-derived measure of disclosed strategic activity comparable in spirit to the thematic scores of the factor layer but with a discrete, interpretable unit. Because the panel is a snapshot universe, firm fixed effects and within-firm comparisons are safer than cross-sectional levels, and the later entry of smaller issuers (\Cref{fig:coverage}) should be modeled explicitly. Two further restrictions follow from this revision: the count must be computed under a stated key and reported with its range (\Cref{sec:speccurve}), and the S\&P~500 cohort must carry a run indicator, since its mentions come from two runs and two 10-K segmenters and its per-issuer counts move by tens of projects depending on which run is counted (\Cref{sec:time}). A firm-year aggregation of mentions also requires the fiscal-year fill exported with the audit outputs, because \texttt{fiscal\_year} is empty for \SPIDfiscalYearNullMentions\ mentions (\Cref{sec:results}). | ||
| 10 | 10 | ||||
| 11 | −\paragraph{Technology adoption and diffusion.} The \texttt{technologies} field and the categories \texttt{ai\_initiative}, \texttt{data\_center}, \texttt{cloud\_migration}, \texttt{automation} and \texttt{erp\_implementation} allow the first-mention date of a technology to be located by issuer, sector and location, in the spirit of \citet{kalyani2025} but from the issuer's own regulatory narrative. The small counts of these categories (a few hundred projects each) and the fact that technologies are free-text tags rather than a controlled vocabulary limit precision; a mapping of tags to a vocabulary is part of the proposed ontology work. | 11 | +\paragraph{Technology adoption and diffusion.} The \texttt{technologies} field and the categories \texttt{ai\_initiative}, \texttt{data\_center}, \texttt{cloud\_migration}, \texttt{automation} and \texttt{erp\_implementation} allow the first-mention date of a technology to be located by issuer, sector and location, in the spirit of \citet{kalyani2025} but from the issuer's own regulatory narrative. The small counts of these categories (a few hundred projects each), the fact that technology tags are free text rather than a controlled vocabulary, and the possibility that up to \SPIDtechRegexEqualPct\% of tag lists were back-filled from the model's description (\Cref{sec:contract}) limit precision; the adoption rates of the Census survey module on advanced technologies \citep{zolas2020} are an external benchmark against which the counts of these categories can be sanity-checked before use, and a mapping of tags to a vocabulary is part of the proposed ontology work. | ||
| 12 | 12 | ||||
| 13 | −\paragraph{Event studies.} Mentions in 8-K filings are dated to the day and are the closest the database comes to an announcement date (74{,}594 mentions). Because the primary document of an 8-K often refers to a press release filed as an exhibit that the pipeline does not read, the first \emph{public} announcement may precede the first mention in the corpus; event dates should be verified against the exhibit or the newswire before use. | 13 | +\paragraph{Event studies.} Mentions in 8-K filings are dated to the day and are the closest the database comes to an announcement date (\SPIDmentionsStoredEightK\ stored mentions), in the tradition of the announcement-level evidence of \citet{mcconnell1985}. Because the primary document of an 8-K often refers to a press release filed as an exhibit that the pipeline does not read, the first \emph{public} announcement may precede the first mention in the corpus; event dates should be verified against the exhibit or the newswire before use, and the \SPIDfinEightKPct\% of 8-K mentions that match the financing pattern should be excluded or relabeled first. | ||
| 14 | 14 | ||||
| 15 | −\paragraph{Project-level panels.} For projects with several mentions, the timeline provides a sequence of status and amount statements that can be used to study announcement-to-completion durations, budget revisions and abandonment, subject to the caveats on status (last statement, not terminal fate) and on merged phases. | 15 | +\paragraph{Project-level panels.} For projects with several mentions, the timeline provides a sequence of status and amount statements that can be used to study announcement-to-completion durations, budget revisions and abandonment, subject to the caveats on status (last statement, not terminal fate; tie order for \SPIDprojectsStatusTie\ projects), on merged phases (\Cref{sec:resolution_rule}) and on amounts whose source wording, currency and scale are not stored (\Cref{sec:amounts}). | ||
| 16 | 16 | ||||
| 17 | \subsection{Identification limits} | 17 | \subsection{Identification limits} | ||
| 18 | 18 | ||||
| 19 | \begin{itemize} | 19 | \begin{itemize} | ||
| 20 | \item \textbf{Disclosure, not activity.} The database measures what issuers chose to say in a regulated document. Disclosure thresholds, materiality judgments, boilerplate habits and litigation risk vary by firm, sector and time; a difference in project counts is a difference in disclosure before it is a difference in investment. | 20 | \item \textbf{Disclosure, not activity.} The database measures what issuers chose to say in a regulated document. Disclosure thresholds, materiality judgments, boilerplate habits and litigation risk vary by firm, sector and time; a difference in project counts is a difference in disclosure before it is a difference in investment. | ||
| 21 | −\item \textbf{Snapshot universe.} Survivorship and backfilling are built in. Comparisons across years that mix cohorts of issuers confound activity with the composition of the observed panel. | 21 | +\item \textbf{Snapshot universe and sampled proxy.} Survivorship and backfilling are built in, and the proxy omits an unknown subset of the smallest index members (\Cref{sec:sample}). Comparisons across years that mix cohorts of issuers confound activity with the composition of the observed panel. | ||
| 22 | −\item \textbf{Measurement error of unknown structure.} Until the human-coded evaluation is run, extraction precision and recall, category accuracy and linkage error are unknown. They are unlikely to be classical: boilerplate-heavy sectors and long MD\&As are more exposed to false positives and to truncation, respectively. | 22 | +\item \textbf{Measurement error of unknown structure.} Until the human-coded evaluation is run, extraction precision and recall, category accuracy and linkage error are unknown. The audits of \Cref{sec:validation} show that the error is unlikely to be classical: long sections are read only in part and in document order, so recall falls with section length and with the position of the passage; financing instruments and boilerplate-heavy items are more exposed to false positives; and the project count is partly an artefact of a key whose dominant branch is a generic token. \citet{ludwig2025} set out the correction that a validation subsample permits; until it exists, the counts are descriptive quantities. | ||
| 23 | −\item \textbf{Amounts.} Disclosed amounts are heterogeneous in type and scope and are never realized expenditure; the project-level maximum is a reading convenience, not an estimate. | 23 | +\item \textbf{Run and cohort.} The S\&P~500 block was extracted in a different run, with a different 10-K parser and an unknown model snapshot. Any design that compares that cohort with the rest of the universe, or that uses 10-K text features, must condition on the run. | ||
| 24 | +\item \textbf{Amounts.} Disclosed amounts are heterogeneous in type and scope, are never realized expenditure, and carry no source wording, currency or scale; the project-level maximum is a reading convenience, not an estimate, and the tail contains values that typing would most likely reject (\Cref{sec:amounts}). | ||||
| 24 | \item \textbf{Descriptive versus causal.} Associations between project disclosures and outcomes (returns, investment, employment) identify the joint determination of disclosure and outcomes, not the effect of the project. Causal designs require a source of variation in the \emph{project} that is orthogonal to the decision to \emph{describe} it, and the database does not supply one. | 25 | \item \textbf{Descriptive versus causal.} Associations between project disclosures and outcomes (returns, investment, employment) identify the joint determination of disclosure and outcomes, not the effect of the project. Causal designs require a source of variation in the \emph{project} that is orthogonal to the decision to \emph{describe} it, and the database does not supply one. | ||
| 25 | \end{itemize} | 26 | \end{itemize} | ||
modified
paper/sections/conclusion.tex
+24 −5
@@ -2,14 +2,33 @@ | |||||
| 2 | \section{Limitations, Reproducibility and Conclusion}\label{sec:conclusion} | 2 | \section{Limitations, Reproducibility and Conclusion}\label{sec:conclusion} | ||
| 3 | % ============================================================================ | 3 | % ============================================================================ | ||
| 4 | 4 | ||||
| 5 | −\subsection{Limitations} | 5 | +\subsection{Limitations and what was not done}\label{sec:limitations} | ||
| 6 | 6 | ||||
| 7 | −Beyond the identification limits above, the current build has concrete technical limitations that bound its use. The issuer universe is a fund-holdings proxy at one date, not licensed Russell membership, and 65 of its issuers are outside the form scope. The parser stores section-level provenance but not character offsets into the raw HTML. The focus windows truncate long sections around lexical cues, so recall on annual reports is unknown and probably lower than on 8-Ks. The resolver is a deterministic key that both merges distinct phases and splits renamed initiatives, and the resulting project count is therefore an artifact of the key as much as a count of initiatives. Amounts carry no type and no realization flag. The 10-Q financial-statement and 8-K sections are read whole, while 10-K Items 3--6 and 8--9B are not read at all. And the ledger cannot yet distinguish an empty result from a failed call. | 7 | +Beyond the identification limits above, the current build has concrete technical limitations that bound its use, and this revision has left several items unexecuted. Each is listed with its reason and, where one can be stated, the direction of the bias it implies. | ||
| 8 | 8 | ||||
| 9 | −\subsection{Reproducibility, ethics and terms} | 9 | +\begin{itemize} | ||
| 10 | +\item \textbf{Universe.} The issuer universe is a fund-holdings proxy at one date that covers \SPIDrussellProxyCoveragePct\% of the index count under representative sampling, not licensed Russell membership; the source holdings and ticker files were not archived, so the first line of \Cref{tab:reconciliation} cannot be reconstructed; and \SPIDissuersWithoutFilings\ issuers are outside the form scope. The omitted members are expected to be the smallest, so cross-sectional statements about small issuers are the least reliable. A licensed constituent history would settle both points and was not available. | ||||
| 11 | +\item \textbf{Parsing.} The parser stores section-level provenance but not character offsets into the raw HTML; tables are flattened in the corpus layer and were removed altogether from the S\&P~500 10-K items prepared in June 2026; \SPIDfullDocumentTenKInvisible\ 10-K filings are invisible to the project layer. Re-parsing the S\&P~500 10-Ks with the corpus-layer segmenter would make the two cohorts comparable and was not done because it would require re-extraction of their sections. | ||||
| 12 | +\item \textbf{Focus windows.} The model saw a median \SPIDmdnaShownMedianPct\% of a submitted 10-K MD\&A and \SPIDmentionsFromLtTwentyFivePct\% of mentions come from sections shown for less than a quarter; recall on long sections is bounded above by the windows (\SPIDrulesFocusRecoveredPct\% of rule-based mentions survive them) and is lowest for passages late in a section, such as the liquidity and capital-resources discussion. Storing the focus spans and widening or re-ordering the windows are pipeline changes that imply new model calls. | ||||
| 13 | +\item \textbf{Resolver.} The resolver is a deterministic key that both merges distinct phases and splits renamed initiatives; \SPIDprojectsKeyedNameTokenPct\% of projects are keyed on a generic name token, the count ranges from \SPIDprojectsKeyTypeDropped\ to \SPIDprojectsKeyStrict\ across reasonable keys, and the representative fields of \SPIDprojectsTieMaxConfidence\ projects depend on tie order. The deterministic tie-breaker is specified and not yet adopted in the stored table (a pipeline change). | ||||
| 14 | +\item \textbf{Extraction contract.} \texttt{amount\_raw} is a formatted float, years and technologies may be back-filled from the model's paraphrase, missing confidences default to \SPIDconfDefault, out-of-vocabulary types default to \texttt{technology\_deployment}, and the raw responses were not stored, so none of these can be undone without re-extraction. Amounts carry no type, currency or scale; the typing pass requires annotation or a second extraction and was not authorised. | ||||
| 15 | +\item \textbf{Ledger.} The ledger cannot distinguish an empty result from a failed call: \SPIDllmFailedJsonDecodeError\ silent failures are known from the October log, \SPIDjuneLlmCallFailed\ from the June log, and the first post-outage batch (\SPIDresumeBatchSections\ sections) is most likely a block of empty answers. The re-submission of its \SPIDresumeBatchCalls\ gate-passing sections was requested and not granted for this revision; the affected issuers' October counts are biased downward until it is run. | ||||
| 16 | +\item \textbf{Two runs.} No June code revision exists, the ledger records no model identifier or prompt hash, and the June snapshot of the model is unknown; the runs are separable only by the backfill timestamp. This cannot be repaired after the fact; the proposed \texttt{extraction\_runs} table prevents a recurrence. | ||||
| 17 | +\item \textbf{Stability.} Temperature zero without a seed is not determinism; the re-submission, prompt-paraphrase and model-substitution protocol of \Cref{sec:val_human} requires model calls that were not authorised, so the stability of the output is unmeasured. | ||||
| 18 | +\item \textbf{Human labels.} The gold standard of \Cref{sec:val_human} is drawn, weighted and documented but not annotated; precision, recall, category accuracy, calibration and linkage error therefore remain unmeasured, and gate~(d) is not met. This is author labour and the main item of the next round. | ||||
| 19 | +\item \textbf{Rule-based baseline.} The agreement rates of \Cref{sec:val_mechanical} are computed on a stratified sample of \SPIDrulesSampleProductive\ productive and \SPIDrulesSampleZero\ zero-mention sections, not on the full ledger; a full-corpus run is a matter of compute time and was deferred. | ||||
| 20 | +\item \textbf{Scope.} The 10-Q sections and 8-K bodies are read whole (subject to the windows) while 10-K Items 3--6 and 8--9B are not read at all, and 8-K exhibits (where press releases live) are not read; the first public announcement of a project may therefore precede its first mention. | ||||
| 21 | +\item \textbf{Release and licence.} Whether the derived tables (\texttt{project\_mentions}, \texttt{projects}, \texttt{project\_timeline}, \texttt{extracted\_sections}) are released as files with the paper, and under which licence, are author decisions not yet taken; until then gate~(f) holds for the exports and exhibit code only. | ||||
| 22 | +\end{itemize} | ||||
| 10 | 23 | ||||
| 11 | −The corpus layer and the project layer are reproducible from the Git repository at revision \texttt{2324b58}, the frozen universe artifact, the download logs and the extraction ledger; the paid extraction step is resumable and idempotent, and the complete run costs roughly 775~USD at October 2026 prices. All numbers in this paper are regenerated by \texttt{code/export\_stats.py} and \texttt{code/make\_tables.py}. The raw filings are public documents retrieved under the SEC's fair-access policy with an identified research client; they are not redistributed, and the repository contains no credentials and no private infrastructure details. Derived datasets (project-level, mention-level, issuer-year and event-time exports) will be released with sample restrictions and a data dictionary once gates (c), (d) and (g) are met. The database does not contain personal data; the only names it stores are those of corporations, places and technologies as they appear in public filings. | 24 | +\subsection{Reproducibility, data availability and terms}\label{sec:reproducibility} | ||
| 25 | + | ||||
| 26 | +\paragraph{Data.} The primary documents are Forms 10-K, 10-Q and 8-K retrieved from the SEC's EDGAR system (public domain) under the fair-access policy, with an identifying user agent, between May 2026 (S\&P~500 issuers) and \SPIDiwvFetchDate\ (remaining issuers); they are not redistributed because of their size (\SPIDrawGb~GB) and can be re-downloaded with \texttt{scripts/download\_filings.py}. The issuer universe merges the Wikipedia S\&P~500 table scraped on \SPIDspWikipediaDate\ (text under the Creative Commons Attribution-ShareAlike 4.0 licence) \citep{wikipediasp500} and the iShares Russell~3000 ETF holdings file published by BlackRock, holdings as of \SPIDiwvHoldingsDate, downloaded on \SPIDiwvFetchDate\ under the iShares website terms of use \citep{iwvholdings2026}; the downloaded holdings file was not archived, while the mapped universe is archived with SHA-256 checksums. The index constituent count is read from the FTSE Russell factsheet of \SPIDfactsheetDate\ \citep{ftserussell3000factsheet2026}. The extraction outputs were generated with the OpenAI API (alias \texttt{gpt-4o-mini} in June 2026; \texttt{gpt-4o-mini-2024-07-18} in October 2026 according to the authors' record) under OpenAI's terms of use; raw API responses were not stored, only the normalized mentions. The derived tables reside in DuckDB on the master node of the authors' cluster and are available from the authors on request; their release as files with the paper is an author decision not yet taken. | ||||
| 27 | + | ||||
| 28 | +\paragraph{Code.} The acquisition, parsing and extraction pipeline is revision \texttt{2324b58} of the private repository \texttt{uqo-research-db.git}; the \SPIDsnapshotFilesN\ files that define the extraction instrument and the resolver are archived with the paper (\texttt{code/audit/spid\_snapshot/}, with SHA-256 checksums). The exhibit code (\texttt{code/export\_stats.py}, \texttt{code/make\_tables.py}), the exports (\texttt{data/}), the audit scripts of this revision (\texttt{code/audit/}) and their outputs (\texttt{results/}) are released with the paper; \texttt{run\_all.sh} regenerates every table, figure and inline number from the exports, and \texttt{tests/} (\SPIDtestsN\ checks) verifies that the exports reconcile. The master-node environment is pinned (\texttt{requirements.lock}, \SPIDrequirementsLockPackages\ packages; Python \SPIDpythonVersion; DuckDB \SPIDduckdbVersion); no licence file has been committed pending the authors' decision (MIT for code and CC BY 4.0 for derived tables are proposed). | ||||
| 29 | + | ||||
| 30 | +\paragraph{Reproducibility.} The exhibits are reproducible from the released materials. The databases are regenerable by the authors from the repository, the frozen universe artefact, the download logs and the extraction ledger; the paid extraction step is resumable and ledgered but not deterministic (\Cref{sec:contract}), and the complete October run cost about \SPIDtotalCostUsd~USD at October 2026 prices. The repository contains no credentials and no private infrastructure details. The database does not contain personal data; the only names it stores are those of corporations, places and technologies as they appear in public filings. | ||||
| 12 | 31 | ||||
| 13 | \subsection{Conclusion} | 32 | \subsection{Conclusion} | ||
| 14 | 33 | ||||
| 15 | −We have documented a reproducible pipeline that turns 527{,}548 SEC filings of 2{,}564 issuers into 308{,}065 evidence-linked statements about 121{,}099 disclosed strategic initiatives, each traceable to an accession number and a filing section, classified into a 21-category taxonomy and organized into timelines. We have been equally explicit about what has \emph{not} been done: the accuracy of extraction, classification and linkage has not been measured against independent human labels; amounts are not typed; events are not separated; the universe is a snapshot. The contribution of this paper is therefore the instrument and its audit rather than a validated measurement. The next step of the research program is the annotation study and the linkage audit described in Section~\ref{sec:validation}; only their results will determine whether the counts reported here can be used as variables, and with what measurement-error model. Until then, SPID is a research database under validation, and we ask that it be cited and used as such. | 34 | +We asked whether distinct strategic projects can be identified, classified, dated, linked and quantified from the SEC filings of a Russell~3000 proxy while preserving source evidence. We have documented a re-runnable, ledgered pipeline that turns \SPIDfilingsTotal\ filings of \SPIDuniverseN\ issuers into \SPIDmentions\ section-linked candidate mentions and \SPIDprojectCountCurrentKey\ candidate projects under one stated key, classified into a \SPIDtaxonomyN-category taxonomy and organized into timelines, and we have audited that pipeline mechanically: the structured fields are found verbatim in their source section for \SPIDgroundAllPct\% of mentions, the model read a median \SPIDmdnaShownMedianPct\% of a 10-K MD\&A, the dominant category co-occurs with the corresponding 8-K item far above its base rate, \SPIDfinMentionsPct\% of mentions are financing instruments rather than projects, and the project count moves between \SPIDprojectsKeyTypeDropped\ and \SPIDprojectsKeyStrict\ with the key. We have been equally explicit about what has \emph{not} been done: the accuracy of extraction, classification and linkage has not been measured against independent human labels; the stability of the instrument has not been tested; amounts are not typed; events are not separated; one post-outage batch is suspect; the universe is a sampled snapshot. The contribution of this paper is therefore the instrument and its audit rather than a validated measurement. The next step of the research program is the annotation of the frame delivered here and the linkage audit; only their results will determine whether the counts reported here can be used as variables, and with what measurement-error model. Until then, SPID is a research database under validation, and we ask that it be cited and used as such. | ||
modified
paper/sections/numbers_pending.tex
+7 −0
@@ -456,3 +456,10 @@ | |||||
| 456 | \providecommand{\SPIDspanZeroN}{67{,}170} % data/duration.csv span 0 | 456 | \providecommand{\SPIDspanZeroN}{67{,}170} % data/duration.csv span 0 | ||
| 457 | \providecommand{\SPIDconfShareHalf}{0.5} % data/confidence_dist.csv 1,611 / 308,065 | 457 | \providecommand{\SPIDconfShareHalf}{0.5} % data/confidence_dist.csv 1,611 / 308,065 | ||
| 458 | \providecommand{\SPIDfinAmountSharePct}{17.1} % 17.8 / 104.2 (results/fp_classes/summary.json) | 458 | \providecommand{\SPIDfinAmountSharePct}{17.1} % 17.8 / 104.2 (results/fp_classes/summary.json) | ||
| 459 | +\providecommand{\SPIDnCostReduction}{6{,}481} % data/projects_by_type.csv cost_reduction n | ||||
| 460 | +\providecommand{\SPIDstabilitySections}{500} % consolidated_review.md §6 item 11 (X1 protocol: 500 sections x 3) | ||||
| 461 | +\providecommand{\SPIDstabilityCalls}{about 2{,}500} % consolidated_review.md §6 item 11 | ||||
| 462 | +\providecommand{\SPIDresubmitSections}{200} % consolidated_review.md §6 item 11 (ADVE-08 ii) | ||||
| 463 | +\providecommand{\SPIDannTextChars}{2{,}000} % results/annotation_frame (worktree CSV text truncation) | ||||
| 464 | +\providecommand{\SPIDannFrameMb}{77} % implemented_changes.md N11 (frames on the node) | ||||
| 465 | +\providecommand{\SPIDunregLongThreshold}{3{,}000} % results/fp_classes/summary.json item2_collision (unreg_gt_3000) | ||||
modified
paper/sections/results.tex
+33 −20
@@ -2,34 +2,34 @@ | |||||
| 2 | \section{Descriptive Patterns}\label{sec:results} | 2 | \section{Descriptive Patterns}\label{sec:results} | ||
| 3 | % ============================================================================ | 3 | % ============================================================================ | ||
| 4 | 4 | ||||
| 5 | −All numbers in this section are produced by \texttt{code/export\_stats.py} and \texttt{code/make\_tables.py} from the merged databases at revision \texttt{2324b58}; none is typed by hand. They describe \emph{disclosed} initiatives as detected by the current pipeline and resolved by the current rule. They are not estimates of real-world project counts, and they have not yet been corrected for the extraction and linkage errors whose measurement is described in Section~\ref{sec:validation}. | 5 | +All tables, figures and inline numbers in this section are produced by \texttt{code/export\_stats.py} (read-only queries on the merged databases at revision \texttt{2324b58}, run on the master node) and \texttt{code/make\_tables.py} (exports and audit outputs to exhibits and macros); the generators contain no data literal, which \texttt{tests/test\_tables.py} checks, and the source and formula of every inline macro are listed in \texttt{results/inline\_numbers.json}. The numbers describe \emph{disclosed} initiatives as detected by the current pipeline and resolved by the current rule. They are not estimates of real-world project counts, they have not been corrected for the extraction and linkage errors whose measurement is described in \Cref{sec:validation}, and the project counts hold under the current key only (\Cref{sec:speccurve}). | ||
| 6 | 6 | ||||
| 7 | \subsection{From sections to projects} | 7 | \subsection{From sections to projects} | ||
| 8 | 8 | ||||
| 9 | −The extraction ledger contains 1{,}123{,}098 sections. The model returned at least one project for 168{,}230 of them (15.0\%), yielding 308{,}065 mentions---0.27 per section on average and 1.83 per productive section. The density differs sharply by form and by item (Table~\ref{tab:items}): the MD\&A items of the 10-K and 10-Q produce 39{,}099 and 99{,}305 mentions respectively, Item~1 (Business) of the 10-K 30{,}492, and the 10-Q financial-statement notes 30{,}159, whereas risk-factor and market-risk items produce almost none despite their length. The 8-K body yields 74{,}594 mentions from 400{,}601 filings. Resolution reduces 308{,}065 mentions to 121{,}099 projects (2.54 mentions per project); 60.2\% of projects rest on a single mention, 17.0\% on two, and 8.2\% on more than five (Table~\ref{tab:status}). | 9 | +The extraction ledger contains \SPIDledgerTotal\ rows, of which \SPIDgatePassTotal\ passed the lexical gate and were submitted to the model (\Cref{sec:focus}). The model returned at least one project for \SPIDproductiveSections\ of them (\SPIDproductiveShareCalls\% of calls, \SPIDproductiveShareLedger\% of rows), yielding \SPIDledgerMentionsReturned\ mentions before identifier de-duplication and \SPIDmentions\ stored mentions---\SPIDmentionsPerCall\ per call and \SPIDmentionsPerProductiveSection\ per productive section. The density differs sharply by form and by item (\Cref{tab:items}): the MD\&A items of the 10-K and 10-Q produce \SPIDtenKmdnaMentions\ and \SPIDtenQmdnaMentions\ mentions respectively, Item~1 (Business) of the 10-K \SPIDtenKbusinessMentions, and the 10-Q financial-statement notes \SPIDtenQfinancialStatementsMentions, whereas risk-factor and market-risk items produce almost none despite their length---a pattern that \Cref{sec:focus} shows to be partly a property of the focus windows rather than of the text. The 8-K body yields \SPIDmentionsStoredEightK\ stored mentions (\SPIDmentionsLedgerEightK\ returned by the model) from \SPIDledgerSectionsEightK\ filings. Resolution reduces \SPIDmentions\ mentions to \SPIDprojectCountCurrentKey\ projects (\SPIDmentionsPerProject\ mentions per project); \SPIDsingleMentionShare\% of projects rest on a single mention, \SPIDtwoMentionShare\% on two, and \SPIDoverFiveMentionShare\% on more than five (\Cref{tab:status}). | ||
| 10 | 10 | ||||
| 11 | \begin{table}[H] | 11 | \begin{table}[H] | ||
| 12 | \centering\small | 12 | \centering\small | ||
| 13 | −\caption{Sections submitted and mentions obtained, by form and item}\label{tab:items} | 13 | +\caption{Ledgered sections, gate-passing calls and mentions obtained, by form and item}\label{tab:items} | ||
| 14 | \begin{threeparttable} | 14 | \begin{threeparttable} | ||
| 15 | \input{tables/tab_items} | 15 | \input{tables/tab_items} | ||
| 16 | \begin{tablenotes}\footnotesize | 16 | \begin{tablenotes}\footnotesize | ||
| 17 | −\item 8-K filings (400{,}601 sections, 74{,}594 mentions) are a single \texttt{full\_body} section each and are omitted from the table. \texttt{full\_body}/\texttt{full\_document} rows are filings in which no Item heading was recognized. | 17 | +\item 8-K filings (\SPIDledgerSectionsEightK\ ledgered sections, \SPIDcallsEightK\ calls, \SPIDmentionsLedgerEightK\ mentions returned) are a single \texttt{full\_body} section each and are omitted from the table. ``Ledgered sections'' are rows of \texttt{extracted\_sections}; ``Gate passed'' recomputes the lexical gate of \Cref{app:prompt} over the ledgered text and counts the sections that reached the model. ``Mentions'' are the counts returned by the model as recorded in the ledger (\SPIDcollisionMentions\ fewer are stored; \Cref{sec:contract}). The 10-K \texttt{full\_body} rows are whole S\&P~500 10-Ks from the June 2026 ingestion in which no Item heading was found (\Cref{sec:segmenters}); the 10-Q \texttt{full\_document} rows are filings in which the corpus-layer parser found no heading. Two rows are dominated by the financing-instrument class of \Cref{sec:construct_def}: \SPIDfinUnregPct\% of the 10-Q \texttt{unregistered\_sales} mentions (Part~II Item~2, where issuer repurchases are tabled) and \SPIDfinEightKPct\% of 8-K mentions match the financing pattern. | ||
| 18 | \end{tablenotes} | 18 | \end{tablenotes} | ||
| 19 | \end{threeparttable} | 19 | \end{threeparttable} | ||
| 20 | \end{table} | 20 | \end{table} | ||
| 21 | 21 | ||||
| 22 | \subsection{Composition by category, sector and status} | 22 | \subsection{Composition by category, sector and status} | ||
| 23 | 23 | ||||
| 24 | −Table~\ref{tab:types} and Figure~\ref{fig:types} show the distribution over the 21 categories. M\&A and post-merger integration dominate (34.5\% of projects), followed by capital-expenditure programs (8.9\%), strategic partnerships (8.2\%), geographic expansion (6.5\%) and plant or site construction (6.3\%). The technology-related categories that motivated the first SPID release---data centers, AI initiatives, cloud migration, automation---are small in count (between 264 and 449 projects each) but carry high disclosed amounts (median 128--166~M USD for data centers and AI initiatives). Two readings of this composition are possible and the data do not discriminate between them: either acquisitions are genuinely the most frequently disclosed discrete initiative in SEC filings, which is plausible given that every material acquisition triggers an 8-K and recurring 10-Q notes; or the extraction contract, which asks for ``concrete initiatives'', is more easily satisfied by acquisitions (named counterparty, date, price) than by internal programs. Disentangling the two is a question for the human-annotation study. | 24 | +\Cref{tab:types} and \Cref{fig:types} show the distribution over the \SPIDtaxonomyN\ categories. M\&A and post-merger integration dominate (\SPIDshareMaIntegration\% of projects), followed by capital-expenditure programs (\SPIDshareCapexProgram\%), strategic partnerships (\SPIDsharePartnership\%), geographic expansion (\SPIDshareGeographicExpansion\%) and plant or site construction (\SPIDsharePlantConstruction\%). These shares include the financing-instrument class: excluding projects whose representative name or description matches the financing pattern, \texttt{capex\_program} falls from \SPIDnCapexProgram\ to \SPIDcapexProjectsExcl\ projects (\SPIDshareCapexProgram\% to \SPIDcapexShareExcl\%), \texttt{cost\_reduction} from \SPIDnCostReduction\ to \SPIDcostProjectsExcl\ and \texttt{ma\_integration} from \SPIDnMaIntegration\ to \SPIDmaProjectsExcl\ (\texttt{results/fp\_classes/table5\_restated\_excl\_financing.csv}); \SPIDfinProjectsRep\ projects (\SPIDfinProjectsRepPct\%) match on their representative fields and \SPIDfinProjectsAny\ have at least one matching mention. The technology-related categories that motivated the first SPID release---data centers, AI initiatives, cloud migration, automation---are small in count (between \SPIDtechCategoriesCountRange\ projects each). Their disclosed amounts are reported in \Cref{tab:types} with the number of projects that carry one and a bootstrap interval for the median: the data-center median of \SPIDmedianAmountDataCenter~M USD rests on \SPIDwithAmountDataCenter\ projects with an amount and the AI median of \SPIDmedianAmountAiInitiative~M on \SPIDwithAmountAiInitiative, so the latter should not be quoted without its $N$. Two readings of the composition are possible and the data do not discriminate between them: either acquisitions are genuinely the most frequently disclosed discrete initiative in SEC filings, which is plausible given that every material acquisition triggers an 8-K and recurring 10-Q notes; or the extraction contract, which asks for ``concrete initiatives'', is more easily satisfied by acquisitions (named counterparty, date, price) than by internal programs. The 8-K item-code consistency check of \Cref{sec:validation} shows that the \texttt{ma\_integration} label co-occurs with the acquisition item at a rate far above its base rate, which supports the first reading for 8-Ks without settling the second. Disentangling the two is a question for the human-annotation study. | ||
| 25 | 25 | ||||
| 26 | \begin{table}[H] | 26 | \begin{table}[H] | ||
| 27 | \centering\small | 27 | \centering\small | ||
| 28 | −\caption{Resolved projects by category}\label{tab:types} | 28 | +\caption{Resolved projects by category, with disclosed amounts and an over-merge indicator}\label{tab:types} | ||
| 29 | \begin{threeparttable} | 29 | \begin{threeparttable} | ||
| 30 | \adjustbox{max width=\textwidth}{\input{tables/tab_types}} | 30 | \adjustbox{max width=\textwidth}{\input{tables/tab_types}} | ||
| 31 | \begin{tablenotes}\footnotesize | 31 | \begin{tablenotes}\footnotesize | ||
| 32 | −\item ``With amount'' counts projects for which at least one mention carries a USD figure; ``median amount'' is the median across those projects of the largest amount cited (see Section~\ref{sec:amounts} on why amounts are never summed). | 32 | +\input{tables/tab_types_note} | ||
| 33 | \end{tablenotes} | 33 | \end{tablenotes} | ||
| 34 | \end{threeparttable} | 34 | \end{threeparttable} | ||
| 35 | \end{table} | 35 | \end{table} | ||
@@ -37,10 +37,10 @@ Table~\ref{tab:types} and Figure~\ref{fig:types} show the distribution over the | |||||
| 37 | \begin{figure}[H] | 37 | \begin{figure}[H] | ||
| 38 | \centering | 38 | \centering | ||
| 39 | \includegraphics[width=0.95\textwidth]{figures/fig_types.pdf} | 39 | \includegraphics[width=0.95\textwidth]{figures/fig_types.pdf} | ||
| 40 | −\caption{Resolved projects by taxonomy category (121{,}099 projects).}\label{fig:types} | 40 | +\caption{Resolved projects by taxonomy category (\SPIDprojectCountCurrentKey\ projects under the current key; counts include the financing-instrument class of \Cref{sec:construct_def}). Source: \texttt{data/projects\_by\_type.csv}; generated by \texttt{code/make\_tables.py}.}\label{fig:types} | ||
| 41 | \end{figure} | 41 | \end{figure} | ||
| 42 | 42 | ||||
| 43 | −By sector (Table~\ref{tab:sectors}), health care (23{,}679 projects, 429 issuers) and industrials (17{,}890) contribute the most projects; utilities, materials, energy and real estate have the most projects per issuer (63--75), and financials the fewest (32), consistent with the intuition that asset-heavy, regulated or resource-extracting firms narrate discrete physical programs while banks describe fewer discrete initiatives. The list of the most ``productive'' issuers (Table~\ref{tab:top_issuers} in the appendix)---uranium and gold miners, airlines, a bitcoin miner, an oilfield-services company---illustrates a second mechanism: issuers whose business \emph{consists} of projects (mines, routes, wells, data-center sites) and whose filings enumerate them one by one. | 43 | +By sector (\Cref{tab:sectors}), health care (\SPIDsectorProjectsHealthCare\ projects, \SPIDsectorIssuersHealthCare\ issuers) and industrials (\SPIDsectorProjectsIndustrials) contribute the most projects; \SPIDsectorTopFourNames\ have the most projects per issuer (\SPIDsectorTopFourPerIssuerRange), and financials the fewest (\SPIDsectorFinancialsPerIssuer), consistent with the intuition that asset-heavy, regulated or resource-extracting firms narrate discrete physical programs while banks describe fewer discrete initiatives---and also consistent with a key that creates one project per site. The list of the most ``productive'' issuers (\Cref{tab:top_issuers} in the appendix)---uranium and gold miners, airlines, a bitcoin miner, an oilfield-services company---is consistent with issuers whose business \emph{consists} of projects (mines, routes, wells, data-center sites) enumerating them one by one; whether it also reflects over-splitting by the key has not been measured. | ||
| 44 | 44 | ||||
| 45 | \begin{table}[H] | 45 | \begin{table}[H] | ||
| 46 | \centering\small | 46 | \centering\small | ||
@@ -48,48 +48,61 @@ By sector (Table~\ref{tab:sectors}), health care (23{,}679 projects, 429 issuers | |||||
| 48 | \begin{threeparttable} | 48 | \begin{threeparttable} | ||
| 49 | \adjustbox{max width=\textwidth}{\input{tables/tab_sectors}} | 49 | \adjustbox{max width=\textwidth}{\input{tables/tab_sectors}} | ||
| 50 | \begin{tablenotes}\footnotesize | 50 | \begin{tablenotes}\footnotesize | ||
| 51 | −\item Sector labels come from the iShares holdings file (Russell proxy) or, for S\&P~500-only issuers, from the Wikipedia GICS table. ``Issuers in universe'' includes issuers without filings. | 51 | +\item Sector labels come from the iShares holdings file (Russell proxy) or, for S\&P~500-only issuers, from the Wikipedia GICS table. ``Issuers in universe'' includes issuers without filings. Projects per issuer are computed over issuers with at least one project; the S\&P~500 issuers were extracted in two runs and with two 10-K segmenters, and their per-issuer counts are not comparable with those of Russell-only issuers without the adjustment of \Cref{sec:time}. | ||
| 52 | \end{tablenotes} | 52 | \end{tablenotes} | ||
| 53 | \end{threeparttable} | 53 | \end{threeparttable} | ||
| 54 | \end{table} | 54 | \end{table} | ||
| 55 | 55 | ||||
| 56 | −The latest status recorded for a project is \texttt{in\_progress} for 41.6\%, \texttt{completed} for 41.0\%, \texttt{planned} for 15.1\% and \texttt{mentioned} for 2.3\% (Table~\ref{tab:status}). Because status is taken from the most recent mention, it is a snapshot of the last thing the issuer said, not the terminal fate of the project: a project that was announced, mentioned once as in progress and then silently abandoned is recorded as in progress. The span between the earliest and latest year cited \emph{in the text} is zero for 55.5\% of projects with known years and exceeds five years for 8.5\%; the brief's distinction between announcement, filing, expected start, observed start, expected and realized completion, and cancellation is not yet implemented as separate dated events (Section~\ref{sec:resolution}). | 56 | +The latest status recorded for a project is \texttt{in\_progress} for \SPIDstatusShareInProgress\%, \texttt{completed} for \SPIDstatusShareCompleted\%, \texttt{planned} for \SPIDstatusSharePlanned\% and \texttt{mentioned} for \SPIDstatusShareMentioned\% (\Cref{tab:status}). Because status is taken from the most recent mention, it is a snapshot of the last thing the issuer said, not the terminal fate of the project: a project that was announced, mentioned once as in progress and then silently abandoned is recorded as in progress. The span between the earliest and latest year cited \emph{in the text} is unknown for \SPIDspanUnknownShareAll\% of projects (\SPIDspanUnknownN); among the \SPIDspanKnownN\ projects with cited years it is zero for \SPIDspanZeroShareKnown\% and exceeds five years for \SPIDspanOverFiveShareKnown\% (\SPIDspanZeroShareAll\% and \SPIDspanOverFiveShareAll\% of all projects). The specification's distinction between announcement, filing, expected start, observed start, expected and realized completion, and cancellation is not yet implemented as separate dated events (\Cref{sec:resolution}). | ||
| 57 | 57 | ||||
| 58 | \begin{table}[H] | 58 | \begin{table}[H] | ||
| 59 | \centering\small | 59 | \centering\small | ||
| 60 | \caption{Status, number of mentions and cited time span of resolved projects}\label{tab:status} | 60 | \caption{Status, number of mentions and cited time span of resolved projects}\label{tab:status} | ||
| 61 | +\begin{threeparttable} | ||||
| 61 | \adjustbox{max width=\textwidth}{\input{tables/tab_status}} | 62 | \adjustbox{max width=\textwidth}{\input{tables/tab_status}} | ||
| 63 | +\begin{tablenotes}\footnotesize | ||||
| 64 | +\item All three panels count the \SPIDprojectCountCurrentKey\ projects of the current key. ``Latest status'' is the status of the most recent mention (ties on the latest filing date, \SPIDprojectsStatusTie\ projects, resolved in execution order). ``Span of cited years'' is the difference between the largest and smallest four-digit year in the mentions' \texttt{years} lists; the \SPIDspanUnknownN\ projects without any cited year (\SPIDspanUnknownShareAll\%) are excluded from the span panel, so its shares are over \SPIDspanKnownN\ projects. Up to \SPIDyearsRegexEqualPct\% of year lists may have been back-filled from the model's description (\Cref{sec:contract}). | ||||
| 65 | +\end{tablenotes} | ||||
| 66 | +\end{threeparttable} | ||||
| 62 | \end{table} | 67 | \end{table} | ||
| 63 | 68 | ||||
| 64 | −\subsection{Amounts, confidence and time}\label{sec:amounts_desc} | 69 | +\subsection{Amounts, confidence and time}\label{sec:time} | ||
| 65 | 70 | ||||
| 66 | −A USD amount is attached to 120{,}240 mentions (39.0\%) and to 56{,}500 projects (46.7\%). Table~\ref{tab:amounts} reports the distribution of the largest amount cited per project: median 128.5~M USD, interquartile range 23--500~M, and a long right tail (P99 of 12.5~B USD). The sum of these maxima across projects is 55.9~trillion USD, a number we report only to make the point that it must \emph{not} be interpreted: it adds transaction values, capex budgets, contract values, financing amounts and estimated costs across projects and years, it double counts initiatives that the resolver split, and it treats a stated budget as if it were spent. Section~\ref{sec:amounts} explains how amounts are stored and what an \texttt{amount\_type} field would need to add. | 71 | +A USD amount is attached to \SPIDamountMentions\ mentions (\SPIDamountMentionsShare\%) and to \SPIDamountProjects\ projects (\SPIDamountProjectsShare\%). \Cref{tab:amounts} reports the distribution of the largest amount cited per project: median \SPIDamountMedianMusd~M USD, interquartile range \SPIDamountPTwoFiveMusd--\SPIDamountPSevenFiveMusd~M, and a long right tail (P99 of \SPIDamountPNineNineBusd~B USD; maximum $\SPIDamountMaxSci$~USD; \SPIDamountGeFiftyBProjects\ projects at or above 50~B USD; \Cref{sec:amounts} on the grounding of the tail). The sum of these maxima across projects is \SPIDamountSumTusd~trillion USD, a number we report only to make the point that it must \emph{not} be interpreted: it adds transaction values, capex budgets, contract values, financing amounts (\SPIDfinAmountSharePct\% of mention amounts match the financing pattern) and estimated costs across projects and years, it double counts initiatives that the resolver split, and it treats a stated budget as if it were spent. \Cref{sec:amounts} explains how amounts are stored and what an \texttt{amount\_type} field would need to add. | ||
| 67 | 72 | ||||
| 68 | \begin{table}[H] | 73 | \begin{table}[H] | ||
| 69 | \centering\small | 74 | \centering\small | ||
| 70 | \caption{Distribution of disclosed amounts and of projects per issuer}\label{tab:amounts} | 75 | \caption{Distribution of disclosed amounts and of projects per issuer}\label{tab:amounts} | ||
| 76 | +\begin{threeparttable} | ||||
| 71 | \adjustbox{max width=\textwidth}{\input{tables/tab_amounts}} | 77 | \adjustbox{max width=\textwidth}{\input{tables/tab_amounts}} | ||
| 78 | +\begin{tablenotes}\footnotesize | ||||
| 79 | +\input{tables/tab_amounts_note} | ||||
| 80 | +\item Projects per issuer are counted over the \SPIDissuersWithProjects\ issuers with at least one project under the current key (maximum \SPIDprojectsPerIssuerMax); the S\&P~500 cohort was extracted in two runs (\Cref{sec:time}). | ||||
| 81 | +\end{tablenotes} | ||||
| 82 | +\end{threeparttable} | ||||
| 72 | \end{table} | 83 | \end{table} | ||
| 73 | 84 | ||||
| 74 | −The self-reported confidence of mentions is concentrated at the top of the scale (Figure~\ref{fig:confidence}; Table~\ref{tab:confidence} in the appendix): 35.2\% of mentions have confidence 1.0, 32.7\% have 0.9, and only 1.6\% fall at or below 0.6. Mentions below 0.45 are discarded at extraction time. The distribution says more about the model's reporting habits than about accuracy; whether a confidence of 0.7 corresponds to a lower precision than a confidence of 1.0 is precisely the calibration question that only labeled data can answer. | 85 | +The self-reported confidence of mentions is concentrated at the top of the scale (\Cref{fig:confidence}; \Cref{tab:confidence} in the appendix): \SPIDconfShareOne\% of mentions have confidence 1.0, \SPIDconfShareNine\% have 0.9, and only \SPIDconfShareLeSix\% fall at or below 0.6. Mentions below \SPIDconfThreshold\ are discarded at extraction time, and the 0.6 bin contains the \SPIDconfExactDefault\ mentions whose confidence the post-processing defaulted because the model returned none (\Cref{sec:contract}). The distribution says more about the model's reporting habits than about accuracy; the only accuracy-related quantity available so far moves the wrong way with confidence---the share of mentions whose structured fields are all found in the source text is \SPIDgroundAllTopBin\% in the 0.9--1.0 bin against \SPIDgroundAllLowerBinsRange\% in the lower bins (\Cref{sec:validation})---and whether a confidence of 0.7 corresponds to a lower precision than a confidence of 1.0 is precisely the calibration question that only labeled data can answer. | ||
| 75 | 86 | ||||
| 76 | \begin{figure}[H] | 87 | \begin{figure}[H] | ||
| 77 | \centering | 88 | \centering | ||
| 78 | \includegraphics[width=0.85\textwidth]{figures/fig_confidence.pdf} | 89 | \includegraphics[width=0.85\textwidth]{figures/fig_confidence.pdf} | ||
| 79 | −\caption{Self-reported extraction confidence of the 308{,}065 mentions (rounded to one decimal; mentions below 0.45 are not stored).}\label{fig:confidence} | 90 | +\caption{Self-reported extraction confidence of the \SPIDmentions\ stored mentions (rounded to one decimal; mentions below \SPIDconfThreshold\ are not stored; the 0.6 bin includes \SPIDconfExactDefault\ defaulted values). Source: \texttt{data/confidence\_dist.csv}; generated by \texttt{code/make\_tables.py}.}\label{fig:confidence} | ||
| 80 | \end{figure} | 91 | \end{figure} | ||
| 81 | 92 | ||||
| 82 | −Over time (Figure~\ref{fig:mentions_year}; Table~\ref{tab:coverage_year}), mentions rise from 11{,}908 in 2010 to 23{,}220 in 2025, almost in proportion to the number of issuers observed (1{,}224 to 2{,}280 with at least one mention), while the number of projects first observed in a year is flat at 6{,}200--8{,}200. The rise is therefore mostly a coverage effect of the snapshot universe, not evidence of an increase in strategic activity; the flat count of new projects per year combined with a growing issuer base implies a declining number of new projects per issuer, which may reflect the lower disclosure intensity of the smaller firms that enter the panel later, or the resolver's tendency to attach later mentions to existing keys. Both explanations are testable with the panel. | 93 | +Over time (\Cref{fig:mentions_year}; \Cref{tab:coverage_year}), mentions rise from \SPIDmentionsFirstYear\ in 2010 to \SPIDmentionsLastFullYear\ in 2025, almost in proportion to the number of issuers observed (\SPIDissuersWithMentionsFirstYear\ to \SPIDissuersWithMentionsLastFullYear\ with at least one mention; \SPIDmentionsPerIssuerFirstYear\ to \SPIDmentionsPerIssuerLastFullYear\ mentions per such issuer), while the number of projects first observed in a year lies between \SPIDfirstSeenRangeInner\ over \SPIDfirstSeenYearMin--\SPIDfirstSeenYearMax. The first and last years are excluded from that range on purpose: 2010 is left-censored, since every initiative already under way when the corpus starts is ``first seen'' then (\SPIDfirstSeenFirstYear\ projects), and 2026 is partial (\SPIDfirstSeenLastYear\ projects to October~27). The rise in mentions is consistent with a coverage effect of the snapshot universe rather than with an increase in disclosed activity: holding the panel fixed at the \SPIDfixedCohortIssuers\ issuers that file in every year from 2010 to 2025, mentions per issuer move from \SPIDfixedCohortMentionsPerIssuerFirst\ to \SPIDfixedCohortMentionsPerIssuerLast\ and newly seen projects per issuer from \SPIDfixedCohortNewPerIssuerFirst\ to \SPIDfixedCohortNewPerIssuerLast\ (\SPIDfixedCohortFirstSeenFirst\ to \SPIDfixedCohortFirstSeenLast\ projects; \texttt{results/timeseries/first\_seen\_fixed\_cohort.csv}). The decline in new projects per issuer is, at least in part, mechanical: under a time-invariant key, later mentions of a serial acquirer or of a multi-phase site attach to an existing project rather than opening a new one. A windowed variant that opens a new episode when a key is silent for more than three (five) years counts \SPIDepisodesKthree\ (\SPIDepisodesKfive) episodes instead of \SPIDprojectCountCurrentKey\ projects, and its first-seen series is reported alongside the current one in \Cref{tab:coverage_year}. Both readings remain testable with the panel; neither has been tested against labels. | ||
| 83 | 94 | ||||
| 84 | \begin{figure}[H] | 95 | \begin{figure}[H] | ||
| 85 | \centering | 96 | \centering | ||
| 86 | \includegraphics[width=\textwidth]{figures/fig_mentions_year.pdf} | 97 | \includegraphics[width=\textwidth]{figures/fig_mentions_year.pdf} | ||
| 87 | −\caption{Project mentions and projects first observed, by filing year (2026 covers January to October 27).}\label{fig:mentions_year} | 98 | +\caption{Project mentions and projects first observed, by filing year, with the per-issuer series in the second panel. The y-axis starts at zero; 2010 (left-censored) and 2026 (January to October~27) are shaded. Source: \texttt{data/mentions\_by\_year.csv}, \texttt{data/projects\_by\_first\_seen\_year.csv}; generated by \texttt{code/make\_tables.py}.}\label{fig:mentions_year} | ||
| 88 | \end{figure} | 99 | \end{figure} | ||
| 89 | 100 | ||||
| 101 | +The two extraction runs add a confound of their own to any comparison between the S\&P~500 cohort and the rest of the universe. Issuers in both lists carry \SPIDcohortBothPerIssuerAll\ projects per issuer against \SPIDcohortRussellOnlyPerIssuer\ for Russell-only issuers, which the earlier version read as a difference in disclosure behaviour; counting only projects touched by an October-run mention, the same issuers carry \SPIDcohortBothPerIssuerOctober. In an issuer-level regression of the project count on an S\&P~500 indicator (\SPIDcohortOlsN\ issuers, HC1 standard errors), the coefficient is \SPIDcohortOlsAll\ (\SPIDcohortOlsAllSe) with all mentions, $\SPIDcohortOlsOctober$ (\SPIDcohortOlsOctoberSe) with October mentions only, and $\SPIDcohortOlsControlled$ (\SPIDcohortOlsControlledSe) once the logarithm of the issuer's ledgered sections is controlled for (\texttt{results/runs/table6\_cohort\_regression.json}). The cohort difference is a difference in run and volume, not evidence about behaviour, and \Cref{app:runs} gives the reconciliation by run and form. | ||||
| 102 | + | ||||
| 90 | \subsection{A provenance example} | 103 | \subsection{A provenance example} | ||
| 91 | 104 | ||||
| 92 | −Table~\ref{tab:provenance} follows one resolved project through its mentions: the Corpus Christi liquefaction expansion of Cheniere Energy (ticker LNG), typed \texttt{plant\_construction}, with eight mentions in 8-K and 10-K filings between April 2014 and March 2022 and a largest cited amount of 11.5~B USD. Each row is traceable: the accession number identifies the document on EDGAR, the section identifier locates the passage in the parsed corpus, and the snippet is the model's one-sentence description, which the contract requires to be grounded in the passage. The example also exposes the resolver's coarseness: the project key for this issuer and type is the location token ``corpus christi'', so the Stage~3 expansion and the earlier trains share one project identifier although Cheniere describes them as distinct phases. A reviewer can see this immediately because the mentions and their snippets are preserved; a user who consumes only the \texttt{projects} table cannot. | 105 | +\Cref{tab:provenance} follows one resolved project through its mentions: the Corpus Christi liquefaction trains of Cheniere Energy (ticker LNG), typed \texttt{plant\_construction}, with \SPIDprovenanceN\ mentions in \SPIDprovenanceForms\ filings between \SPIDprovenanceFirst\ and \SPIDprovenanceLast\ and a largest cited amount of \SPIDprovenanceMaxAmountBusd~B USD. Each row is traceable: the accession number identifies the document on EDGAR, the section identifier locates the passage in the parsed corpus, and the snippet is the model's one-sentence description, which the contract requires to be grounded in the passage but which is not itself checked (\Cref{sec:contract}). The example also exposes the resolver's coarseness. The project key for this issuer and type is the location string ``san patricio county, texas'', so the Stage~3 expansion and the earlier trains share one project identifier although Cheniere describes them as distinct phases; the representative name, ``Corpus Christi Liquefaction Stage~3 Project'', is that of the highest-confidence mention, filed in \SPIDprovenanceNameYear, and is applied retroactively to mentions from 2014 that describe the first trains; and that representative is one admissible choice among tied mentions (\Cref{sec:resolution_rule}). Under the strict key the same mentions form \SPIDchenierClusterStrict\ project each; under the issuer-and-type key they join \SPIDchenierClusterCikType\ mentions. A reviewer can see this because the mentions and their snippets are preserved; a user who consumes only the \texttt{projects} table cannot. | ||
| 93 | 106 | ||||
| 94 | \begin{table}[H] | 107 | \begin{table}[H] | ||
| 95 | \centering\footnotesize | 108 | \centering\footnotesize | ||
@@ -97,11 +110,11 @@ Table~\ref{tab:provenance} follows one resolved project through its mentions: th | |||||
| 97 | \begin{threeparttable} | 110 | \begin{threeparttable} | ||
| 98 | \adjustbox{max width=\textwidth}{\input{tables/tab_provenance}} | 111 | \adjustbox{max width=\textwidth}{\input{tables/tab_provenance}} | ||
| 99 | \begin{tablenotes}\footnotesize | 112 | \begin{tablenotes}\footnotesize | ||
| 100 | −\item Snippets are the first 170 characters of the model-generated description stored in \texttt{project\_timeline.snippet}; the full source section is retrievable through \texttt{project\_mentions.section\_id}. Amounts are shown only for mentions that cite one. | 113 | +\item All \SPIDprovenanceN\ rows are Form 8-K mentions (\texttt{data/example\_provenance.csv}). Snippets are the first characters of the model-generated description stored in \texttt{project\_timeline.snippet}; the full source section is retrievable through \texttt{project\_mentions.section\_id}. Amounts are shown only for mentions that cite one. The representative name shown in the caption of the project is the \SPIDprovenanceNameYear\ mention's name; it is not a label of the 2014 mentions. | ||
| 101 | \end{tablenotes} | 114 | \end{tablenotes} | ||
| 102 | \end{threeparttable} | 115 | \end{threeparttable} | ||
| 103 | \end{table} | 116 | \end{table} | ||
| 104 | 117 | ||||
| 105 | \subsection{Knowledge graph and research-ready exports} | 118 | \subsection{Knowledge graph and research-ready exports} | ||
| 106 | 119 | ||||
| 107 | −The resolved projects are projected into a graph of 183{,}472 nodes (2{,}492 companies, 121{,}099 projects, 14{,}508 locations, 8{,}551 technologies, 36{,}822 partners) and 204{,}456 edges (\texttt{owns}, \texttt{located\_in}, \texttt{uses}; Table~\ref{tab:graph} in the appendix); partner nodes are currently created without edges, a known gap. Each project also has a 384-dimensional sentence embedding of its name, type, description and technologies, which supports semantic search through the companion web application. Alongside the project layer, the corpus layer publishes the firm-year and firm-quarter panels of thematic factor scores (32{,}852 and 128{,}043 rows) used in earlier UQO work on AI disclosure, now covering the full universe. | 120 | +The resolved projects are projected into a graph of \SPIDkgNodes\ nodes (\SPIDkgNodesCompany\ companies, \SPIDkgNodesProject\ projects, \SPIDkgNodesLocation\ locations, \SPIDkgNodesTechnology\ technologies, \SPIDkgNodesPartner\ partners) and \SPIDkgEdges\ edges (\texttt{owns}, \texttt{located\_in}, \texttt{uses}; \Cref{tab:graph} in the appendix); partner nodes are currently created without edges, a known gap, and \texttt{uses} edges inherit the technology tags of which up to \SPIDtechRegexEqualPct\% may have been back-filled from the description (\Cref{sec:contract}). Each project also has a \SPIDembeddingDim-dimensional sentence embedding of its name, type, description and technologies, which supports semantic search through the companion web application. Alongside the project layer, the corpus layer publishes the firm-year and firm-quarter panels of thematic factor scores (\SPIDannualPanelRows\ and \SPIDquarterlyPanelRows\ rows for \SPIDannualPanelFirms\ issuers with filings) used in the sibling UQO working paper on AI disclosure \citep{kammoun2026ai}, now covering the full universe. A firm-year aggregation of the \emph{mentions}, by contrast, is not yet available as stated in the earlier version: \texttt{fiscal\_year} is empty for \SPIDfiscalYearNullMentions\ mentions (all \SPIDbackfilledMentions\ June mentions and \SPIDfiscalYearNullOctTenQ\ October 10-Q mentions among others) because the corresponding filings carry no fiscal year, and a derived fill from the filing date is exported with the audit outputs but not stored (\Cref{app:runs}). | ||
modified
paper/sections/validation.tex
+45 −23
@@ -2,49 +2,71 @@ | |||||
| 2 | \section{Validation Framework and Current Audit}\label{sec:validation} | 2 | \section{Validation Framework and Current Audit}\label{sec:validation} | ||
| 3 | % ============================================================================ | 3 | % ============================================================================ | ||
| 4 | 4 | ||||
| 5 | −The aim of identifying ``all projects'' of an issuer is a \emph{coverage target}, not an assumption. Only projects that appear in the captured sources, survive parsing and gating, are recognized by the model and are correctly linked are observable, and each of these steps loses or distorts some. This section specifies how those losses are to be measured and reports what has been measured so far. The headline is simple: the \emph{coverage} and \emph{integrity} components are implemented and quantified; the \emph{accuracy} components---which require independent human labels---are specified but not yet executed. Table~\ref{tab:gates} summarizes the status of the acceptance gates. | 5 | +The aim of identifying ``all projects'' of an issuer is a \emph{coverage target}, not an assumption. Only projects that appear in the captured sources, survive parsing and gating, fall inside the focus windows, are recognized by the model and are correctly linked are observable, and each of these steps loses or distorts some. In the framework of \citet{ludwig2025}, the output of \Cref{sec:construct} is a model-generated measurement whose error structure is unknown until a validation subsample with independent labels identifies it; until then it can describe the corpus but cannot serve as a regressor without an explicit measurement-error model. This section specifies how the losses are to be measured and reports what has been measured. The headline is in three parts: the \emph{coverage} and \emph{integrity} components are implemented and quantified (\Cref{sec:val_coverage}); the \emph{mechanical} half of accuracy---containment of the structured fields in the source text, what the model was shown, consistency with an external classification, agreement with a rule-based baseline, and the size of known false-positive classes---was computed in this revision for every mention or section without any new model call (\Cref{sec:val_mechanical}); and the components that require independent human labels are specified, with the sampling frame drawn and weighted, but not executed (\Cref{sec:val_human}). \Cref{tab:gates} summarizes the status of the acceptance gates. | ||
| 6 | 6 | ||||
| 7 | −\subsection{Coverage and integrity checks (implemented)} | 7 | +\subsection{Coverage and integrity checks (implemented)}\label{sec:val_coverage} | ||
| 8 | 8 | ||||
| 9 | \begin{itemize} | 9 | \begin{itemize} | ||
| 10 | −\item \textbf{Universe reconciliation} (Table~\ref{tab:reconciliation}): every input line of the Russell proxy is accounted for as matched, dropped-as-non-company, or unmatched; the 65 issuers without filings are listed by name and reason. | 10 | +\item \textbf{Universe reconciliation} (\Cref{tab:reconciliation}): the mapped universe and its two derived inputs are archived with checksums, and every mapped line is accounted for as matched or unmatched with the \SPIDrussellUnmatched\ unmatched tickers listed; the \SPIDissuersWithoutFilings\ issuers without filings are listed by name and reason. The source holdings file and the SEC ticker file are not archived and the lines dropped before mapping are not listed, so the check is partial (gate~(a)). | ||
| 11 | −\item \textbf{Acquisition completeness:} for every issuer the SEC submissions index is enumerated in full (recent block and archive blocks) and every retained filing is logged; download failures are counted per issuer and were zero in the final run. Filing counts by form and year are published (Table~\ref{tab:coverage_year}). | 11 | +\item \textbf{Acquisition completeness:} for every issuer the SEC submissions index is enumerated in full (recent block and archive blocks) and every retained filing is logged; \SPIDfilingsMissingOnDisk\ filings logged as downloaded were not found on disk at parse time and are counted as failures. Filing counts by form and year are published (\Cref{tab:coverage_year}). | ||
| 12 | −\item \textbf{Parsing coverage:} 16 of 527{,}548 primary documents could not be parsed (Table~\ref{tab:filings}); filings without recognizable Item headings are kept as a single section rather than dropped. | 12 | +\item \textbf{Parsing and scope coverage:} \SPIDfilingsNotParsed\ of \SPIDfilingsTotal\ primary documents could not be parsed (\Cref{tab:filings}); filings without recognizable Item headings are kept as a single section rather than dropped, which for 10-Ks places them outside the submitted scope (\SPIDfullDocumentTenKInvisible\ filings invisible to the project layer); the 10-K funnel from parsed rows to ledgered rows reconciles exactly (\Cref{sec:segmenters}). | ||
| 13 | −\item \textbf{Extraction ledger:} every section submitted to the model is recorded; the ledger was audited after the credit-exhaustion incident and all sections processed during the outage were re-submitted. The ledger does not yet record a per-call status, so an undetected API failure would still appear as ``no project''; the run logs (retried errors, JSON parse failures: five in the June run, one in the October run) are the only current evidence that such failures were rare. | 13 | +\item \textbf{Extraction ledger:} every ledgered section is recorded with its mention count and timestamp; the true number of calls (\SPIDgatePassTotal) is recovered by recomputing the gate; the outage rows removed from the ledger reconcile with the run log and no row remains inside the outage gap (\Cref{sec:ledger}). The ledger does not record a per-call status, so an undetected API failure still appears as ``no project'': the run logs show \SPIDllmFailedJsonDecodeError\ such failures outside the outage in October and \SPIDjuneLlmCallFailed\ in June, and the first batch after the restart (\SPIDresumeBatchSections\ sections, \SPIDresumeBatchProductive\ productive against about \SPIDresumeBatchExpectedProductive\ expected) is flagged as suspect pending a re-submission that was not authorised for this revision. | ||
| 14 | −\item \textbf{Referential integrity:} foreign keys from chunks to sections to filings to companies are enforced by the database; the merge of ten shards is append-only on globally unique keys and was checked for duplicate accession numbers and section identifiers (none). | 14 | +\item \textbf{Two runs:} the June and October blocks can be separated only by the backfill timestamp; the ledger stores no model identifier or prompt hash, and no June code revision exists (\Cref{sec:versioning}). The inter-revision diff of the extractor and parser files is printed in \Cref{app:runs}. | ||
| 15 | −\item \textbf{Reproducibility of exhibits:} every table and figure of this paper is regenerated from the databases by two scripts in \texttt{code/}; the numbers can be compared against \texttt{data/stats.json}. | 15 | +\item \textbf{Referential integrity:} foreign keys from chunks to sections to filings to companies are enforced by the database; the merge of ten shards is append-only on globally unique keys and was checked for duplicate accession numbers and section identifiers (none). Mention identifiers, however, collide for \SPIDcollisionMentions\ mentions in \SPIDcollisionSections\ sections (\Cref{sec:contract}). | ||
| 16 | +\item \textbf{Reproducibility of exhibits:} every table, figure and inline number of this paper is regenerated from the exported data and audit outputs by two scripts in \texttt{code/}; \SPIDtestsN\ automated checks (\texttt{tests/}) verify that the exports reconcile with each other, that regeneration is byte-identical, and that the generator contains no data literal. The exports themselves were regenerated read-only on the master node in this revision and found byte-identical to the committed ones except for the ordering of tied rows in two files. | ||||
| 16 | \end{itemize} | 17 | \end{itemize} | ||
| 17 | 18 | ||||
| 18 | −\subsection{Accuracy evaluation (specified, not executed)} | 19 | +\subsection{Mechanical accuracy audits (implemented in this revision)}\label{sec:val_mechanical} | ||
| 19 | 20 | ||||
| 20 | −\paragraph{Gold standard.} A stratified sample of sections will be drawn from the ledger with strata defined by form (10-K, 10-Q, 8-K), item, issuer size (fund weight tercile), sector, filing year (2010--2014, 2015--2019, 2020--2026) and disclosure length. The sample will include sections that produced mentions and, in equal number, randomly selected sections that produced none, so that both false positives and false negatives can be estimated. Two annotators will independently code each section: for each initiative present, its name, category (from the 21-code taxonomy), status, cited years, location, partners, amount and amount type, plus the five-way label of Section~\ref{sec:construct} (project, generic strategy, third-party, uncertain, none). Disagreements will be adjudicated by the third team member; inter-annotator agreement (Cohen's $\kappa$ for presence and category, exact-match rates for fields) will be reported. The sample will be split into a development half, on which prompt and threshold changes may be tuned, and a locked holdout half, used once per release. | 21 | +\paragraph{Field-level grounding.} The prompt requires that numbers and locations appear in the text, and the structured fields are literal tokens, so their containment in the stored section text can be tested for every mention. For all \SPIDmentions\ mentions joined to their section text (\texttt{results/grounding/}; \SPIDgroundBootstrapReps\ bootstrap replicates clustered by section), every listed year is found in \SPIDgroundYearsPct\% of mentions with years (\SPIDgroundYearsCiLow--\SPIDgroundYearsCiHigh), every listed partner in \SPIDgroundPartnersPct\%, the amount as a number (in any of the usual renderings) in \SPIDgroundAmountNumberPct\% and with a currency token within forty characters in \SPIDgroundAmountCurrencyPct\%, every listed location in \SPIDgroundLocationsPct\% (10-K \SPIDgroundLocationsTenK\%, 10-Q \SPIDgroundLocationsTenQ\%, 8-K \SPIDgroundLocationsEightK\%), and every technology tag in \SPIDgroundTechPct\%; at the level of individual elements the rates are \SPIDgroundElemYears\% for years, \SPIDgroundElemPartners\% for partners, \SPIDgroundElemLocations\% for locations and \SPIDgroundElemTech\% for technologies. All structured fields of a mention are found in \SPIDgroundAllPct\% of the \SPIDgroundAllN\ mentions that carry at least one (10-K \SPIDgroundAllTenK\%, 10-Q \SPIDgroundAllTenQ\%, 8-K \SPIDgroundAllEightK\%; June block \SPIDgroundAllJune\%, October \SPIDgroundAllOctober\%), with no difference between filings before and after the model's training cut-off beyond one percentage point. Three readings of these rates are needed. First, they are rates of \emph{transcription}, not of correctness: a year, partner or amount can be copied faithfully from a passage that describes a counterparty's initiative, a hypothetical, or a financing instrument, so grounding is necessary for a correct mention and not sufficient. Second, the lower rates for locations and technologies reflect normalization by the model (``Texas'' for ``Austin, TX''; ``cloud computing'' for ``AWS'') as much as invention; the element-level rate for locations is \SPIDgroundElemLocations\%. Third, \SPIDyearsAbsentFromText\ mentions carry a year absent from the section, and the technology and year lists equal the regex output on the model's own description for up to \SPIDtechRegexEqualPct\% and \SPIDyearsRegexEqualPct\% of mentions, so for those fields ``grounded'' may mean ``copied from the paraphrase'' (\Cref{sec:contract}). The full breakdown by field, form and item is in \Cref{tab:grounding}. | ||
| 21 | 22 | ||||
| 22 | −\paragraph{Metrics.} On the holdout: candidate-gate recall (share of annotated projects whose section passed the gate); project-detection precision, recall and F1 at the mention level; category macro- and micro-F1; field accuracy for amount, years and location; the abstention rate (sections where the model returned an empty list) conditional on annotated content; and confidence calibration (precision by confidence bin, with bootstrap intervals). The same metrics will be computed for the rule-based extractor and, if budget allows, for a semantic-retrieval baseline, so that the marginal value of the LLM step is documented. | 23 | +\paragraph{What the model was shown.} \Cref{sec:focus} reports the shown fraction of every ledgered section. The recall of the pipeline on long sections is bounded from above by what the windows contain: on the over-budget sections the rule-based extractor of the code base recovers \SPIDrulesFocusRecoveredPct\% of its full-text mentions from the focused text (\SPIDrulesFocusMdnaPct\% in the 10-K MD\&A). Gate~(e) is therefore reported as met at passage level, not at section level. | ||
| 23 | 24 | ||||
| 24 | −\paragraph{Linkage audit.} From the resolved projects, a sample stratified by number of mentions (1, 2--5, 6--10, more than 10) and by category will be reviewed to label each project cluster as correct, over-merged (contains distinct initiatives) or under-merged (a duplicate of another cluster of the same issuer). Link precision and recall and the error costs of false merges versus missed links will be reported, with particular attention to long-running programs, subsidiaries and near-duplicate names. | 25 | +\paragraph{External consistency with 8-K item codes.} Form 8-K bodies name the items they report under, and \SPIDeightKItemCodePct\% of the \SPIDledgerSectionsEightK\ 8-K sections carry at least one item code in their text, which gives an external classification at no cost (\texttt{results/benchmark\_8k/}). Among 8-Ks filed under Item~2.01 (completion of an acquisition or disposition; \SPIDbenchTwoOhOneN\ sections) the model returned at least one \texttt{ma\_integration} mention for \SPIDbenchMaGivenTwoOhOne\ (\SPIDbenchMaGivenTwoOhOneCiLow--\SPIDbenchMaGivenTwoOhOneCiHigh) of sections, against \SPIDbenchMaGivenEightOhOne\ under Item~8.01 (other events) and \SPIDbenchMaGivenFiveOhTwo\ under Item~5.02 (officer changes) and a base rate of \SPIDbenchMaBase; conversely \SPIDbenchItemsGivenMa\ of 8-K \texttt{ma\_integration} mentions sit in a filing under Item~2.01 or 1.01. Under Item~2.05 (exit or disposal costs; \SPIDbenchTwoOhFiveN\ sections) the model returned a \texttt{cost\_reduction} mention for \SPIDbenchCostGivenTwoOhFive\ (\SPIDbenchCostGivenTwoOhFiveCiLow--\SPIDbenchCostGivenTwoOhFiveCiHigh) of sections, a lift of about \SPIDbenchCostLift\ over the base rate, and \SPIDbenchTwoOhFiveGivenCost\ of \texttt{cost\_reduction} mentions sit under that item; Item~1.01 (material agreements) carries a \texttt{partnership} mention in \SPIDbenchPartnershipGivenOneOhOne\ of cases against a base rate of \SPIDbenchPartnershipBase. These are consistency rates with an administrative classification of the \emph{filing}, not precision or recall of the \emph{mention}: an Item~2.01 filing without an M\&A mention may describe a disposition, and an Item~8.01 filing with one may be a genuine acquisition announced under a general item. They show that the dominant category is not an artefact of the extractor's vocabulary; they do not measure its accuracy. | ||
| 25 | 26 | ||||
| 26 | −\paragraph{Temporal test.} For a sample of projects with at least three mentions, the status and amount reconstructed using only mentions filed on or before each date will be compared with the full-information values, to verify that no later information leaks into earlier states and to document how often status reverses. | 27 | +\paragraph{Agreement with a rule-based baseline.} The code base contains a rule-based extractor (\texttt{rules.py}) that needs no model call. Run on a seeded, stratified sample of \SPIDrulesSampleProductive\ productive and \SPIDrulesSampleZero\ zero-mention ledger sections with inverse-probability weights (\texttt{results/rules/}), the rules flag at least one candidate in \SPIDrulesGivenLlmPct\% of the sections where the model returned a mention (10-K \SPIDrulesGivenLlmTenK\%, 10-Q \SPIDrulesGivenLlmTenQ\%, 8-K \SPIDrulesGivenLlmEightK\%; June \SPIDrulesGivenLlmJune\%, October \SPIDrulesGivenLlmOctober\%), while the model returns a mention in \SPIDllmGivenRulesPct\% of the sections the rules flag; the rules flag \SPIDrulesFlagPct\% of ledgered sections against \SPIDproductiveShareLedger\% for the model, section-level agreement is \SPIDrulesAgreementPct\% (model-only \SPIDrulesLlmOnlyPct\%, rules-only \SPIDrulesOnlyPct\%), and on co-detected sections the model's category is among the rule categories in \SPIDrulesTypeOverlapPct\% of cases (\texttt{ma\_integration} \SPIDrulesTypeMa\%, \texttt{drug\_pipeline} \SPIDrulesTypeDrug\%, \texttt{cost\_reduction} \SPIDrulesTypeCost\%, \texttt{capex\_program} \SPIDrulesTypeCapex\%, \texttt{plant\_construction} \SPIDrulesTypePlant\%, \texttt{geographic\_expansion} \SPIDrulesTypeGeo\%). Neither extractor is a ground truth, so this is a measure of the marginal information in the model step---it abstains on most of what the lexical rules flag and rarely fires where they do not---rather than of accuracy. A full-corpus run of the rules is deferred; the sample gives roughly half a percentage point of precision on the rates. | ||
| 27 | 28 | ||||
| 28 | −\paragraph{Missingness.} Zero-project issuer-years will be cross-tabulated by sector, size and filing count to distinguish ``no disclosure'' from ``not processed''; the former is informative, the latter is a data gap, and the \texttt{coverage\_status} field (complete for the defined scope, partial, failed, not observed) proposed for the issuer-year export will carry that distinction. | 29 | +\paragraph{Known false-positive classes.} The financing-instrument pattern of \Cref{sec:construct_def} matches \SPIDfinMentionsPct\% of mentions, concentrated in Part~II Item~2 of the 10-Q and in 8-K bodies, and \Cref{sec:results} restates the category counts without them. A related parser check finds that the 10-Q Part~II Item~2 section (median \SPIDunregMedianWords\ words) is longer than \SPIDunregLongThreshold\ words in \SPIDunregGtThreeK\ of \SPIDitemTwoPairs\ filings (\SPIDunregGtThreeKPct\%), \SPIDswapCandidates\ of which have an MD\&A shorter than 500 words and are candidates for a Part~I/Part~II ``Item~2'' swap by the segmenter; \SPIDunregGtThreeKMentions\ mentions come from those sections. Both classes are detected by pattern, not by annotation, so their sizes are indicative, and the annotation protocol labels them explicitly. | ||
| 30 | + | ||||
| 31 | +\paragraph{Resolver sensitivity.} The dependence of the project count on the key and on the confidence threshold is reported in \Cref{sec:speccurve}; the over-merge indicators (distinct names and partners per project, amounts taken from a different counterparty) are in \Cref{sec:resolution_rule}. | ||||
| 32 | + | ||||
| 33 | +\subsection{Accuracy evaluation against human labels (frame drawn, labels not produced)}\label{sec:val_human} | ||||
| 34 | + | ||||
| 35 | +\paragraph{Sampling frame.} The unit is the ledgered section, that is, exactly the text block the extractor received a verdict for. The frame was drawn with a fixed seed (\SPIDannSeed) from the two populations of the ledger (\texttt{results/annotation\_frame/}; protocol in \texttt{docs/annotation\_protocol.md}). From the \SPIDannPopProductive\ productive sections, \SPIDannStrata\ strata are defined by item group, run (June block or October), the shown-fraction bin of \Cref{sec:focus}, the confidence bin of the section's best mention and the category group; allocation is proportional with a floor of \SPIDannFloor\ per stratum and a \SPIDannOversample-fold oversampling of the confidence bin at or below 0.7, which gives $n = \SPIDannNProductive$ (the floor dominates; a nested core subsample of \SPIDannCore\ drawn proportionally within the frame is flagged for a first pass). From the \SPIDannPopZero\ zero-mention sections, \SPIDannNZero\ are drawn with inclusion probability proportional to the gate verdict and to the logarithm of section length, so that gate-rejected sections are represented (\SPIDannZeroGatePass\ gate-passing, \SPIDannZeroGateReject\ gate-rejected) and false negatives can be estimated with inverse-probability weights; the earlier version's equal-count draw without weights would have biased recall upward. The frame is split into a development half (\SPIDannDev\ sections), on which prompt and threshold changes may be tuned, and a locked holdout half (\SPIDannHoldout), used once per release. The design effects are \SPIDannDeffProductive\ (productive) and \SPIDannDeffZero\ (zero-mention); the 95\% half-widths are $\pm$\SPIDannHalfWidthPrecision\ percentage points for a precision near 0.5 and $\pm$\SPIDannHalfWidthFn\ points for a false-negative rate near 5\%, and the minimum detectable difference in the false-negative rate between gate-passing and gate-rejected strata is \SPIDannMde\ points. The full-text frames, the text the model was shown, and the sealed model output are stored on the master node (\SPIDannFrameMb~MB); the worktree copies carry the text truncated to \SPIDannTextChars\ characters. | ||||
| 36 | + | ||||
| 37 | +\paragraph{Labels and annotators.} Two annotators will independently code each section without seeing the model output (it is unsealed only for the mention-level comparison): at the section level, whether the section describes at least one project and how many (labels S1--S2); at the mention level after unsealing, for each initiative its presence, category (from the \SPIDtaxonomyN-code taxonomy), status, cited years, location, partners, amount and amount type, plus the six-way label of \Cref{sec:construct_def} (project, generic strategy, third-party, financing instrument, uncertain, none) and a boilerplate flag (M1--M4); and at the project level, whether a cluster is correct, over-merged or under-merged (L1). Disagreements will be adjudicated by the third team member; inter-annotator agreement will be reported as Krippendorff's $\alpha$ and prevalence-adjusted $\kappa$ for presence and category, and as exact-match rates for fields. The annotators are the authors; no label has been produced at the time of writing, and no label generated by a language model will be reported as a human label. | ||||
| 38 | + | ||||
| 39 | +\paragraph{Metrics.} On the holdout, with the design weights: candidate-gate recall (share of annotated projects whose section passed the gate); project-detection precision, recall and F1 at the mention level, by form, item and shown-fraction bin; category macro- and micro-F1; field accuracy for amount, years and location; the abstention rate (sections where the model returned an empty list) conditional on annotated content; the share of mentions in each false-positive class; and confidence calibration (precision by confidence bin, with bootstrap intervals, the 0.6 bin reported with and without the defaulted values). The same metrics will be computed for the rule-based extractor, so that the marginal value of the model step is documented, and the estimated error structure will be the input to the measurement-error correction that \citet{ludwig2025} describe. | ||||
| 40 | + | ||||
| 41 | +\paragraph{Linkage audit.} From the resolved projects, a sample stratified by number of mentions (1, 2--5, 6--10, more than 10) and by category will be reviewed to label each project cluster as correct, over-merged (contains distinct initiatives) or under-merged (a duplicate of another cluster of the same issuer). Link precision and recall and the error costs of false merges versus missed links will be reported, with particular attention to long-running programs, subsidiaries, near-duplicate names and the generic-token keys of \Cref{sec:resolution_rule}. | ||||
| 42 | + | ||||
| 43 | +\paragraph{Stability of the instrument.} A protocol of \SPIDstabilitySections\ sections re-submitted three times, once with a paraphrased prompt and once to a substitute model (\SPIDstabilityCalls\ calls), and a re-submission of the \SPIDresumeBatchCalls\ gate-passing sections of the suspect post-outage batch, are specified; both require new model calls, which were not authorised for this revision, and their results are absent from this paper. | ||||
| 44 | + | ||||
| 45 | +\paragraph{Temporal test and missingness.} For a sample of projects with at least three mentions, the status and amount reconstructed using only mentions filed on or before each date will be compared with the full-information values, to verify that no later information leaks into earlier states and to document how often status reverses. Zero-project issuer-years will be cross-tabulated by sector, size and filing count to distinguish ``no disclosure'' from ``not processed''; the former is informative, the latter is a data gap, and the \texttt{coverage\_status} field (complete for the defined scope, partial, failed, not observed) proposed for the issuer-year export will carry that distinction. | ||||
| 29 | 46 | ||||
| 30 | \subsection{Acceptance gates} | 47 | \subsection{Acceptance gates} | ||
| 31 | 48 | ||||
| 32 | \begin{table}[H] | 49 | \begin{table}[H] | ||
| 33 | \centering\small | 50 | \centering\small | ||
| 34 | −\caption{Acceptance gates for a research-ready release and their status at revision \texttt{2324b58}}\label{tab:gates} | 51 | +\caption{Acceptance gates for a research-ready release and their status at revision \texttt{2324b58} after the Stage D audits}\label{tab:gates} | ||
| 35 | −\begin{tabularx}{\textwidth}{lXl} | 52 | +\begin{threeparttable} | ||
| 53 | +\begin{tabularx}{\textwidth}{@{}l>{\raggedright\arraybackslash}p{5.6cm}>{\raggedright\arraybackslash}X@{}} | ||||
| 36 | \toprule | 54 | \toprule | ||
| 37 | Gate & Requirement & Status \\ | 55 | Gate & Requirement & Status \\ | ||
| 38 | \midrule | 56 | \midrule | ||
| 39 | −(a) & Target issuer universe and time window frozen and auditable & Met (snapshot design; file hashes pending) \\ | 57 | +(a) & Target issuer universe and time window frozen and auditable & Partially met: snapshot design; mapped universe and derived inputs archived with SHA-256 checksums; the source holdings file and SEC ticker file are not archived and the lines dropped before mapping are not listed \\ | ||
| 40 | −(b) & Every published project links to at least one verifiable primary-source passage & Met at section level; character spans not stored \\ | 58 | +(b) & Every published project links to at least one verifiable primary-source passage & Met at section level; structured fields mechanically checked against the section text (\SPIDgroundAllPct\% fully grounded); description fidelity not checkable; character spans not stored \\ | ||
| 41 | −(c) & Duplicate, temporal and amount rules documented and tested & Documented; unit tests partial; no event or amount-type tables \\ | 59 | +(c) & Duplicate, temporal and amount rules documented and tested & Documented; the pipeline snapshot contains no test; the exhibit layer has \SPIDtestsN\ automated checks that do not test the pipeline; no event or amount-type tables; representative fields depend on tie order \\ | ||
| 42 | −(d) & Extraction and matching accuracy evaluated on independent human labels & Not met (protocol specified) \\ | 60 | +(d) & Extraction and matching accuracy evaluated on independent human labels & Not met: protocol revised, frame drawn and weighted ($n = \SPIDannNProductive + \SPIDannNZero$), annotation pending (author task) \\ | ||
| 43 | −(e) & Issuer/form/year coverage and failures quantified & Met \\ | 61 | +(e) & Issuer/form/year coverage and failures quantified & Met at passage level: coverage, parse and scope failures, gate rejections, outage, silent failures and shown fractions quantified; one post-outage batch flagged as suspect \\ | ||
| 44 | −(f) & Every empirical result regenerable from a pinned build & Met for this paper (\texttt{code/}, revision \texttt{2324b58}) \\ | 62 | +(f) & Every empirical result regenerable from a pinned build & Exhibits and exports regenerable from the released code and data (\texttt{run\_all.sh}, lock files); the databases are regenerable by the authors on the master node; the extraction step is re-runnable and ledgered, not deterministic; release of the derived tables is an author decision not yet taken \\ | ||
| 45 | (g) & All coauthors have reviewed scientific claims and authorship & Pending \\ | 63 | (g) & All coauthors have reviewed scientific claims and authorship & Pending \\ | ||
| 46 | \bottomrule | 64 | \bottomrule | ||
| 47 | \end{tabularx} | 65 | \end{tabularx} | ||
| 66 | +\begin{tablenotes}\footnotesize | ||||
| 67 | +\item Gates are those of the research-design specification (\Cref{sec:sample}). ``Met at passage level'' means that the quantity is established for the text the model was actually shown, not for the whole section. Items marked as author tasks or decisions are listed in \Cref{sec:limitations}. | ||||
| 68 | +\end{tablenotes} | ||||
| 69 | +\end{threeparttable} | ||||
| 48 | \end{table} | 70 | \end{table} | ||
| 49 | 71 | ||||
| 50 | −Until gates (c), (d) and (g) are met, the database is a \emph{research database under validation}, and any use of its counts or amounts in empirical work should treat extraction and linkage error as measurement error of unknown magnitude. | 72 | +Until gates (c), (d) and (g) are met, the database is a \emph{research database under validation}, and any use of its counts or amounts in empirical work should treat extraction and linkage error as measurement error of unknown structure, with the audits above as the only available bounds. | ||
| 51 | 73 | ||||