% Owner: Agent 10 (Stage E writing). Every generated exhibit is \input exactly once in the prose at the anchor % where it is discussed; please do not add a second \input of the same float (duplicate labels). — 2026-11-05 % ============================================================================ \section{Architecture: Acquisition, Parsing and the Distributed Build}\label{sec:architecture} % ============================================================================ \subsection{Overview} \Cref{fig:pipeline} summarizes the pipeline. Two layers are distinguished throughout. The \emph{corpus layer} (code in \texttt{src/} and \texttt{scripts/}) acquires and parses filings, chunks them, embeds the chunks and computes thematic factor scores; it is universe-agnostic and produces a DuckDB database (\texttt{edgar\_research.duckdb}, \SPIDcorpusDbGb~GB) plus Parquet exports (\SPIDcorpusParquetGb~GB of section text). The \emph{project layer} (code in \texttt{project-db/spid/}) reads the Parquet exports through views, runs candidate gating and LLM extraction, resolves mentions into projects, builds a knowledge graph and project embeddings, and persists everything in a second, much smaller DuckDB file (\texttt{spid.duckdb}, \SPIDspidDbGb~GB). The separation matters for re-runs: the paid step (LLM extraction) depends only on the frozen Parquet exports and on a ledger of processed sections, so it can be resumed or audited independently of the corpus build. One exception to the separation, documented in \Cref{sec:segmenters}, is that the 10-K items of the S\&P~500 block were prepared in June 2026 by a parser internal to the project layer and are reused as such. \begin{figure}[H] \centering \begin{tikzpicture}[node distance=4mm and 6mm, every node/.style={font=\footnotesize}, box/.style={draw, rounded corners=2pt, align=center, minimum height=8mm, text width=28mm, fill=blue!4}, art/.style={draw, align=center, minimum height=7mm, text width=28mm, fill=yellow!10, font=\scriptsize}, arr/.style={-{Latex[length=2mm]}, thick}] \node[box] (u) {Universe snapshot\\ \scriptsize S\&P~500 $\cup$ IWV $\to$ CIK}; \node[box, right=of u] (dl) {SEC acquisition\\ \scriptsize submissions index, primary docs, \SPIDsecRateLimit\ req/s per IP}; \node[box, right=of dl] (p) {Parsing\\ \scriptsize HTML $\to$ text, Item segmentation, chunks}; \node[box, right=of p] (e) {Embeddings \& factors\\ \scriptsize MiniLM \SPIDembeddingDim-d, \SPIDfactorAnchors\ anchors}; \node[art, below=of u] (ua) {\texttt{universe.json}\\ \texttt{shards.json}}; \node[art, below=of dl] (da) {\texttt{download\_log.jsonl}\\ raw HTML on \SPIDacqNodes\ nodes}; \node[art, below=of p] (pa) {\texttt{filings}, \texttt{filing\_sections}\\ \texttt{filing\_chunks} (Parquet)}; \node[art, below=of e] (ea) {\texttt{embeddings/}\\ \texttt{chunk\_scores}, panels}; \node[box, below=14mm of da, fill=red!5] (x) {Candidate gate + LLM extraction\\ \scriptsize regex gate, focus windows, JSON-object mode}; \node[box, right=of x, fill=red!5] (r) {Resolution \& timeline\\ \scriptsize key = CIK $|$ type $|$ location/name}; \node[box, right=of r, fill=red!5] (g) {Graph, project embeddings,\\ analytics, API}; \node[art, below=of x] (xa) {\texttt{project\_mentions} \SPIDmentions\\ \texttt{extracted\_sections} \SPIDledgerTotal}; \node[art, below=of r] (ra) {\texttt{projects} \SPIDprojectCountCurrentKey\\ \texttt{project\_timeline}}; \node[art, below=of g] (ga) {\texttt{kg\_nodes/edges}\\ \texttt{project\_embeddings}}; \draw[arr] (u)--(dl); \draw[arr] (dl)--(p); \draw[arr] (p)--(e); \draw[arr] (u)--(ua); \draw[arr] (dl)--(da); \draw[arr] (p)--(pa); \draw[arr] (e)--(ea); \draw[arr] (pa.south) |- ++(0,-5mm) -| (x.north); \draw[arr] (x)--(r); \draw[arr] (r)--(g); \draw[arr] (x)--(xa); \draw[arr] (r)--(ra); \draw[arr] (g)--(ga); \node[above=1mm of dl, font=\scriptsize\itshape] {Corpus layer (\texttt{src/}, \texttt{scripts/})}; \node[above=1mm of r, yshift=-1mm, font=\scriptsize\itshape] {}; \node[left=1mm of x, font=\scriptsize\itshape, text width=16mm, align=right] {Project layer (\texttt{spid/})}; \end{tikzpicture} \caption{Pipeline architecture. Rounded boxes are processing steps; rectangles are persisted artefacts. The project layer reads the Parquet exports of the corpus layer through read-only views and never modifies them; the 10-K items of the S\&P~500 block are the exception described in \Cref{sec:segmenters}. The diagram is drawn by hand (TikZ), not generated from data; the counts shown are the inline macros of \texttt{tables/numbers.tex}.}\label{fig:pipeline} \end{figure} \subsection{SEC acquisition} The SEC's fair-access policy allows \SPIDsecRateLimit\ requests per second per client and asks for a descriptive \texttt{User-Agent} with a contact address \citep{secfairaccess}. Both constraints shaped the acquisition layer (\texttt{src/acquisition/bulk\_downloader.py}). Requests are issued by a pool of threads behind a single token bucket; the bucket rate is a parameter, and when several processes share one public IP address the orchestrator divides \SPIDacqGatewayDivisor\ requests per second among them. HTTP 403, 429 and 503 responses trigger a \SPIDcooldownSeconds-second cool-down for the whole process rather than a per-request retry, so that a throttling signal is never answered with a burst. Downloads are idempotent: a filing is skipped when its file exists, and every completed download is appended to a JSON-lines log carrying the accession number, form, filing and report dates, primary document name, size and local path. A trusted-log option lets a node skip filings that an earlier corpus already contains without re-fetching them. The SEC submissions header of each issuer (name, SIC, state of incorporation, exchanges, fiscal-year end, former names) is saved alongside, and feeds the \texttt{companies} table. The October 2026 build used eleven machines: \SPIDacqNodes\ Apple-silicon nodes of the UQO MacLustr cluster behind one residential IP address, and one Linux server in an OVHcloud data center in Beauharnois, Québec, behind a second address. Issuers were assigned to nodes in proportion to CPU cores; half of each node's issuers were downloaded by the node itself (\SPIDacqNodeRate\ requests per second each) and half by the external server (\SPIDacqGatewayRate\ request per second per shard), then shipped to the node over SSH. A first allocation by contiguous CIK ranges proved badly unbalanced---old CIKs are large filers with many documents each---and was replaced after one hour by an interleaved allocation of the not-yet-started issuers. According to the run notes of the research log, the acquisition of \SPIDacqNewDocs\ new documents took \SPIDacqHours\ hours at a sustained \SPIDacqReqPerSec\ requests per second with no fair-access violation recorded; the orchestrator logs behind these figures are not archived with the paper, so they are reported as run facts rather than as exported statistics. Two filings are logged as downloaded but were not found on disk at parse time (\Cref{tab:filings}); in substance they are download failures. \subsection{Document processing: two parsing paths}\label{sec:segmenters} The corpus layer cleans each primary document with \texttt{lxml}: \texttt{script}, \texttt{style}, \texttt{meta} and \texttt{link} elements are removed, tables are flattened to tab-separated text, lists are kept as line items, whitespace is normalized and files above \SPIDmaxFileMb~MB are skipped. The segmenter then locates ``Item~\emph{n}'' and ``Part~I/II'' headings by regular expression: 10-Ks are split into up to \SPIDtenKsectionsMax\ sections (Items 1 to 9B), 10-Qs into \SPIDtenQsectionsN\ (Part~I Items 1--4, Part~II Items 1, 1A, 2), and 8-Ks are kept as a single \texttt{full\_body} section. A filing in which no heading is found is kept as one \texttt{full\_document} section rather than dropped, so that the foreign-key chain from chunks to sections to filings is never broken; for 10-Ks such sections lie outside the submitted scope (\SPIDfullDocumentTenK\ filings), for 10-Qs they are submitted (\SPIDfullDocumentTenQLedgered\ of \SPIDfullDocumentTenQ). Sections are chunked into \SPIDchunkWords-word windows with a \SPIDchunkOverlap-word overlap for the embedding and factor layers (\SPIDchunks\ chunks). Provenance is preserved at the section level: each section row stores its item number, order, word and character counts, and each chunk stores its character offsets within the section. Character-level offsets of sections within the raw HTML are \emph{not} stored, a limitation recorded in the audit. The 10-K items of the S\&P~500 block did not go through this path. They were produced in June 2026 by the project layer's own ingestion module (\texttt{spid/ingest.py}: BeautifulSoup, every \texttt{} element removed, a separate Item regular expression, the longest instance of each Item kept) and stored in a legacy table (\texttt{spid\_sections}, \SPIDlegacyTenKRows\ rows for \SPIDlegacyTenKFilings\ filings); the October project layer keeps the June text for every 10-K already present in that table and reads the corpus-layer sections only for the others. The two segmenters produce measurably different text. On the \SPIDsegPairs\ item pairs available in both tables, the legacy MD\&A has a median of \SPIDlegacyMdnaWords\ words against \SPIDparsedMdnaWords\ for the same item in the corpus layer, \SPIDlegacyMdnaDollar\ dollar signs on average against \SPIDparsedMdnaDollar, and \SPIDlegacyMdnaNumeric\ numeric tokens against \SPIDparsedMdnaNumeric; \SPIDlegacyFewerMoneyPct\% of MD\&A pairs carry fewer money tokens in the legacy text, and the legacy Properties item is less than half the length of its counterpart in \SPIDpropertiesLtHalfPct\% of pairs (\texttt{results/runs/segmenter\_item\_comparison.csv}). Capital-expenditure tables, which the legacy ingestion removed, are among the passages a capex extractor would most want to see. The \SPIDtenKfullBodyRows\ rows labeled \texttt{full\_body} in \Cref{tab:items} are also legacy rows: whole 10-Ks (median \SPIDfullBodyMedianWords\ words) in which the June ingestion found no Item heading, squeezed through the focus windows of \Cref{sec:construct}; the corpus layer has proper items for \SPIDfullBodyWithParsedItems\ of these filings, but those \SPIDfullBodyUnsubmittedSections\ in-scope sections were never submitted because the project layer defers to the legacy table. The earlier version of this paper attributed the \SPIDtenKfullBodyRows\ rows to the October segmenter; that was wrong. The practical consequence is that the S\&P~500 cohort's 10-K text and the Russell-only cohort's 10-K text were prepared by different cleaners, which \Cref{sec:results} and \Cref{app:runs} take into account when the two cohorts are compared. \Cref{tab:sections} reports the outcome by form. Two coverage statistics bound what the extractor could see: \SPIDfilingsNotParsedPct\% of filings were not parsed, and \SPIDfullDocumentTenKInvisible\ 10-K filings have neither recognized items nor a legacy row and are invisible to the project layer. The 10-K funnel reconciles exactly: \SPIDtenKparsedRows\ parsed 10-K rows minus \SPIDtenKoutOfScopeRows\ out-of-scope items, minus \SPIDtenKlegacyAccessionRows\ rows of filings covered by the legacy table, minus \SPIDtenKunderThirtyRows\ rows under \SPIDminWords\ words, gives \SPIDtenKparsedLedgered\ corpus-layer rows, which with the \SPIDlegacyTenKRows\ legacy rows equal the \SPIDledgerSectionsOneZeroK\ 10-K rows of the ledger. % exhibit generated by code/make_tables.py (self-contained float) \input{tables/tab_sections} \subsection{Distributed build and merge} Parsing, chunking and embedding ran on every node against the node's own shard, using the same code revision and a local DuckDB file; each node then exported its tables to Parquet. The master node (M3U96b, which also holds the original S\&P~500 corpus) pulled the shards and merged them with an append-only rule: a row is inserted only if its primary key (accession number, section or chunk UUID) is absent, so that re-running the merge cannot create duplicates. The merged database holds \SPIDuniverseN\ issuers, \SPIDfilingsTotal\ filings, \SPIDsectionsTotal\ sections, \SPIDchunks\ chunks and as many \SPIDembeddingDim-dimensional embeddings. The run notes record a parsing throughput of about \SPIDparseFilingsPerSec\ filings per second per node (\SPIDparseMinutesRange\ minutes per shard), \SPIDembedMinutesRange\ minutes per node for the embeddings on Apple-silicon GPUs, and \SPIDmergeMinutes\ minutes for the merge and the factor computation; like the acquisition figures these are run facts, not exported statistics. The orchestration script (\texttt{scripts/cluster/orchestrate.py}) is stage-based and idempotent: every stage polls marker files on the nodes and can be restarted without side effects. Private infrastructure details (addresses, credentials, keys) are kept outside the repository and outside this paper. \subsection{Versioning and the two extraction runs}\label{sec:versioning} The repository is versioned on a private Git server. The revision used for the October 2026 build is \texttt{2324b58} (October~28, 2026). The repository holds \SPIDrepoCommits\ commits, all dated October~27--28, 2026, so no revision of the code exists for the June 2026 run; the earlier version of this paper and the June methodological study \citep{spid2026} cited \texttt{df7905d} as the June revision, but that commit is dated October~27, 2026. What can be established is file-system evidence: the extractor modules (\texttt{llm.py}, \texttt{rules.py}, \texttt{base.py}, \texttt{taxonomy.py}, \texttt{resolve.py}, \texttt{ingest.py}) carry modification times of \SPIDextractorMtimeDate, before the June run, while the driver files (\texttt{config.py}, \texttt{pipeline.py}, \texttt{store.py}) were edited on October~27, 2026 before the first commit, so their June versions are unrecoverable. The diff between the two October revisions on the extractor, parser and driver files is \SPIDdiffStoreLines\ lines in \texttt{store.py} (a DuckDB~2.0 Arrow fallback), and the diff from the first commit adds \SPIDdiffParserLines\ lines in \texttt{filing\_parser.py}; both patches are reproduced in \Cref{app:runs}. The prompt text is therefore unchanged across the two runs as far as the file system can show, which is weaker than a per-run digest. The extraction ledger does not help here. Its \texttt{backend} column holds the constant string \texttt{\SPIDledgerBackendValue} for every row of both runs; neither a model identifier nor a prompt, schema or taxonomy hash is stored, and the run logs contain no model string either. The October run is reported by the authors as having used the snapshot \texttt{gpt-4o-mini-2024-07-18}, and the June run the unpinned alias \texttt{gpt-4o-mini}; the first identity rests on the authors' record rather than on an archived artefact, and the June snapshot is unknown. The June block was not re-extracted in October: its \SPIDbackfilledRows\ ledger rows were written at a single timestamp (\SPIDbackfillTimestamp) when the ledger table was created, without the \SPIDminWords-word filter that the October run applied (\SPIDbackfilledUnderThirtyWords\ rows are shorter), which is why the ledger's time range begins on October~27 and why the two runs can be separated only by that timestamp. The proposed \texttt{extraction\_runs} table (\Cref{app:dictionary}) records, per run, the model snapshot, the \texttt{seed} and \texttt{system\_fingerprint} returned by the API, and the prompt and taxonomy hashes; none of these exists for the current build.