spb/firc0001_uqo

1% ============================================================================2\section{Architecture: Acquisition, Parsing and the Distributed Build}\label{sec:architecture}3% ============================================================================45\subsection{Overview}67Figure~\ref{fig:pipeline} summarizes the pipeline. Two layers are distinguished throughout. The \emph{corpus layer} (code in \texttt{src/} and \texttt{scripts/}) acquires and parses filings, chunks them, embeds the chunks and computes thematic factor scores; it is universe-agnostic and produces a DuckDB database (\texttt{edgar\_research.duckdb}, 68~GB) plus Parquet exports. The \emph{project layer} (code in \texttt{project-db/spid/}) reads the Parquet exports through views, runs candidate gating and LLM extraction, resolves mentions into projects, builds a knowledge graph and project embeddings, and persists everything in a second, much smaller DuckDB file (\texttt{spid.duckdb}, 3.2~GB). The separation matters for reproducibility: the expensive and paid step (LLM extraction) depends only on the frozen Parquet exports and on a ledger of processed sections, so it can be re-run, resumed or audited independently of the corpus build.89\begin{figure}[H]10\centering11\begin{tikzpicture}[node distance=4mm and 6mm, every node/.style={font=\footnotesize},12  box/.style={draw, rounded corners=2pt, align=center, minimum height=8mm, text width=28mm, fill=blue!4},13  art/.style={draw, align=center, minimum height=7mm, text width=28mm, fill=yellow!10, font=\scriptsize},14  arr/.style={-{Latex[length=2mm]}, thick}]15\node[box] (u) {Universe snapshot\\ \scriptsize S\&P~500 $\cup$ IWV $\to$ CIK};16\node[box, right=of u] (dl) {SEC acquisition\\ \scriptsize submissions index, primary docs, 10 req/s per IP};17\node[box, right=of dl] (p) {Parsing\\ \scriptsize HTML $\to$ text, Item segmentation, chunks};18\node[box, right=of p] (e) {Embeddings \& factors\\ \scriptsize MiniLM 384-d, 8 anchors};19\node[art, below=of u] (ua) {\texttt{universe.json}\\ \texttt{shards.json}};20\node[art, below=of dl] (da) {\texttt{download\_log.*.jsonl}\\ raw HTML on 10 nodes};21\node[art, below=of p] (pa) {\texttt{filings}, \texttt{filing\_sections}\\ \texttt{filing\_chunks} (Parquet)};22\node[art, below=of e] (ea) {\texttt{embeddings/} 16~GB\\ \texttt{chunk\_scores}, panels};23\node[box, below=14mm of da, fill=red!5] (x) {Candidate gate + LLM extraction\\ \scriptsize regex gate, focus windows, JSON schema};24\node[box, right=of x, fill=red!5] (r) {Resolution \& timeline\\ \scriptsize key = CIK $|$ type $|$ location/name};25\node[box, right=of r, fill=red!5] (g) {Graph, project embeddings,\\ analytics, API};26\node[art, below=of x] (xa) {\texttt{project\_mentions} 308{,}065\\ \texttt{extracted\_sections} 1.12~M};27\node[art, below=of r] (ra) {\texttt{projects} 121{,}099\\ \texttt{project\_timeline}};28\node[art, below=of g] (ga) {\texttt{kg\_nodes/edges}\\ \texttt{project\_embeddings}};29\draw[arr] (u)--(dl); \draw[arr] (dl)--(p); \draw[arr] (p)--(e);30\draw[arr] (u)--(ua); \draw[arr] (dl)--(da); \draw[arr] (p)--(pa); \draw[arr] (e)--(ea);31\draw[arr] (pa.south) |- ++(0,-5mm) -| (x.north);32\draw[arr] (x)--(r); \draw[arr] (r)--(g);33\draw[arr] (x)--(xa); \draw[arr] (r)--(ra); \draw[arr] (g)--(ga);34\node[above=1mm of dl, font=\scriptsize\itshape] {Corpus layer (\texttt{src/}, \texttt{scripts/})};35\node[above=1mm of r, yshift=-1mm, font=\scriptsize\itshape] {};36\node[left=1mm of x, font=\scriptsize\itshape, text width=16mm, align=right] {Project layer (\texttt{spid/})};37\end{tikzpicture}38\caption{Pipeline architecture. Rounded boxes are processing steps; rectangles are persisted artifacts. The project layer reads the Parquet exports of the corpus layer through read-only views and never modifies them.}\label{fig:pipeline}39\end{figure}4041\subsection{SEC acquisition}4243The SEC's fair-access policy allows ten requests per second per client and asks for a descriptive \texttt{User-Agent} with a contact address \citep{secfairaccess}. Both constraints shaped the acquisition layer (\texttt{src/acquisition/bulk\_downloader.py}). Requests are issued by a pool of threads behind a single token bucket; the bucket rate is a parameter, and when several processes share one public IP address the orchestrator divides nine requests per second among them. HTTP 403, 429 and 503 responses trigger a 65-second cool-down for the whole process rather than a per-request retry, so that a throttling signal is never answered with a burst. Downloads are idempotent: a filing is skipped when its file exists, and every completed download is appended to a JSON-lines log carrying the accession number, form, filing and report dates, primary document name, size and local path. A trusted-log option lets a node skip filings that an earlier corpus already contains without re-fetching them. The SEC submissions header of each issuer (name, SIC, state of incorporation, exchanges, fiscal-year end, former names) is saved alongside, and feeds the \texttt{companies} table.4445The October 2026 build used eleven machines: ten Apple-silicon nodes of the UQO MacLustr cluster behind one residential IP address, and one Linux server in an OVHcloud data center in Beauharnois, Québec, behind a second address. Issuers were assigned to nodes in proportion to CPU cores; half of each node's issuers were downloaded by the node itself (0.9 requests per second each) and half by the external server (one request per second per shard), then shipped to the node over SSH. A first allocation by contiguous CIK ranges proved badly unbalanced---old CIKs are large filers with up to 220 documents each---and was replaced after one hour by an interleaved allocation of the not-yet-started issuers. The whole acquisition (about 400{,}000 new documents) took seven hours at a sustained 17 requests per second, with no fair-access violation recorded.4647\subsection{Document processing}4849Each primary document is cleaned with \texttt{lxml}: \texttt{script}, \texttt{style}, \texttt{meta} and \texttt{link} elements are removed, tables are flattened to tab-separated text, lists are kept as line items, whitespace is normalized and files above 50~MB are skipped. The segmenter then locates ``Item~\emph{n}'' and ``Part~I/II'' headings by regular expression: 10-Ks are split into up to fourteen sections (Items 1 to 9B), 10-Qs into seven (Part~I Items 1--4, Part~II Items 1, 1A, 2), and 8-Ks are kept as a single \texttt{full\_body} section. A filing in which no heading is found is kept as one \texttt{full\_document} section rather than dropped, so that the foreign-key chain from chunks to sections to filings is never broken. Sections are chunked into 512-word windows with a 64-word overlap for the embedding and factor layers (10.72 million chunks). Provenance is preserved at the section level: each section row stores its item number, order, word and character counts, and each chunk stores its character offsets within the section. Character-level offsets of sections within the raw HTML are \emph{not} stored, a limitation recorded in the audit.5051Table~\ref{tab:sections} reports the outcome. The 10-K sections are long (median 875 words, mean 5{,}551) and the 10-Q sections somewhat shorter; 8-K bodies are short (median 575 words). Two coverage statistics deserve emphasis because they bound what the extractor could see: 0.003\% of filings failed parsing, and the segmenter's reliance on textual headings means that a 10-K whose items are rendered as images or in unusual markup collapses into a single \texttt{full\_document} section (1{,}230 such 10-K sections were sent to the extractor).5253\begin{table}[H]54\centering\small55\caption{Parsed sections and sections submitted to the extractor, by form}\label{tab:sections}56\begin{threeparttable}57\adjustbox{max width=\textwidth}{\input{tables/tab_sections}}58\begin{tablenotes}\footnotesize59\item ``Sections sent to the extractor'' counts rows of the \texttt{extracted\_sections} ledger: sections with at least 30 words, restricted for 10-Ks to Items 1, 1A, 2, 7 and 7A (see Table~\ref{tab:items}). Parsed 10-K sections also include Items 3--6, 8, 9--9B, which are not submitted.60\end{tablenotes}61\end{threeparttable}62\end{table}6364\subsection{Distributed build and merge}6566Parsing, chunking and embedding ran on every node against the node's own shard, using the same code revision and a local DuckDB file; each node then exported its tables to Parquet. The master node (M3U96b, which also holds the original S\&P~500 corpus) pulled the shards and merged them with an append-only rule: a row is inserted only if its primary key (accession number, section or chunk UUID) is absent, so that re-running the merge cannot create duplicates. The merged database holds 2{,}564 issuers, 527{,}548 filings, 1{,}342{,}283 sections, 10{,}720{,}890 chunks and as many 384-dimensional embeddings. Parsing throughput was about 100 filings per second per node (five to thirteen minutes per shard); embeddings took 15--40 minutes per node on Apple-silicon GPUs; the merge and the factor computation took seven minutes. The orchestration script (\texttt{scripts/cluster/orchestrate.py}) is stage-based and idempotent: every stage polls marker files on the nodes and can be restarted without side effects. Private infrastructure details (addresses, credentials, keys) are kept outside the repository and outside this paper.6768\subsection{Versioning}6970The repository is versioned in a private Git server; the revision used for the build described here is \texttt{2324b58} (October~28, 2026), which supersedes revision \texttt{df7905d} examined in the June 2026 methodological study \citep{spid2026}. Model and prompt versions are recorded in the extraction ledger as the backend name; the OpenAI model identifier returned by the API during the October run was \texttt{gpt-4o-mini-2024-07-18}. A hash of the prompt and of the taxonomy is \emph{not} yet stored per run---an item on the audit list---so prompt identity is currently guaranteed by the Git revision rather than by a per-run digest.71