spb/firc0001_uqo

1% Owner: Agent 10 (Stage E writing). Every generated exhibit is \input exactly once in the prose at the anchor2% where it is discussed; please do not add a second \input of the same float (duplicate labels). — 2026-11-053% ============================================================================4\section{Validation Framework and Current Audit}\label{sec:validation}5% ============================================================================67The aim of identifying ``all projects'' of an issuer is a \emph{coverage target}, not an assumption. Only projects that appear in the captured sources, survive parsing and gating, fall inside the focus windows, are recognized by the model and are correctly linked are observable, and each of these steps loses or distorts some. In the framework of \citet{ludwig2025}, the output of \Cref{sec:construct} is a model-generated measurement whose error structure is unknown until a validation subsample with independent labels identifies it; until then it can describe the corpus but cannot serve as a regressor without an explicit measurement-error model. This section specifies how the losses are to be measured and reports what has been measured. The headline is in three parts: the \emph{coverage} and \emph{integrity} components are implemented and quantified (\Cref{sec:val_coverage}); the \emph{mechanical} half of accuracy---containment of the structured fields in the source text, what the model was shown, consistency with an external classification, agreement with a rule-based baseline, and the size of known false-positive classes---was computed in this revision for every mention or section without any new model call (\Cref{sec:val_mechanical}); and the components that require independent human labels are specified, with the sampling frame drawn and weighted, but not executed (\Cref{sec:val_human}). \Cref{tab:gates} summarizes the status of the acceptance gates.89\subsection{Coverage and integrity checks (implemented)}\label{sec:val_coverage}1011\begin{itemize}12\item \textbf{Universe reconciliation} (\Cref{tab:reconciliation}): the mapped universe and its two derived inputs are archived with checksums, and every mapped line is accounted for as matched or unmatched with the \SPIDrussellUnmatched\ unmatched tickers listed; the \SPIDissuersWithoutFilings\ issuers without filings are listed by name and reason. The source holdings file and the SEC ticker file are not archived and the lines dropped before mapping are not listed, so the check is partial (gate~(a)).13\item \textbf{Acquisition completeness:} for every issuer the SEC submissions index is enumerated in full (recent block and archive blocks) and every retained filing is logged; \SPIDfilingsMissingOnDisk\ filings logged as downloaded were not found on disk at parse time and are counted as failures. Filing counts by form and year are published (\Cref{tab:coverage_year}).14\item \textbf{Parsing and scope coverage:} \SPIDfilingsNotParsed\ of \SPIDfilingsTotal\ primary documents could not be parsed (\Cref{tab:filings}); filings without recognizable Item headings are kept as a single section rather than dropped, which for 10-Ks places them outside the submitted scope (\SPIDfullDocumentTenKInvisible\ filings invisible to the project layer); the 10-K funnel from parsed rows to ledgered rows reconciles exactly (\Cref{sec:segmenters}).15\item \textbf{Extraction ledger:} every ledgered section is recorded with its mention count and timestamp; the true number of calls (\SPIDgatePassTotal) is recovered by recomputing the gate; the outage rows removed from the ledger reconcile with the run log and no row remains inside the outage gap (\Cref{sec:ledger}). The ledger does not record a per-call status, so an undetected API failure still appears as ``no project'': the run logs show \SPIDllmFailedJsonDecodeError\ such failures outside the outage in October and \SPIDjuneLlmCallFailed\ in June, and the first batch after the restart (\SPIDresumeBatchSections\ sections, \SPIDresumeBatchProductive\ productive against about \SPIDresumeBatchExpectedProductive\ expected) is flagged as suspect pending a re-submission that was not authorised for this revision.16\item \textbf{Two runs:} the June and October blocks can be separated only by the backfill timestamp; the ledger stores no model identifier or prompt hash, and no June code revision exists (\Cref{sec:versioning}). The inter-revision diff of the extractor and parser files is printed in \Cref{app:runs}.17\item \textbf{Referential integrity:} foreign keys from chunks to sections to filings to companies are enforced by the database; the merge of ten shards is append-only on globally unique keys and was checked for duplicate accession numbers and section identifiers (none). Mention identifiers, however, collide for \SPIDcollisionMentions\ mentions in \SPIDcollisionSections\ sections (\Cref{sec:contract}).18\item \textbf{Reproducibility of exhibits:} every table, figure and inline number of this paper is regenerated from the exported data and audit outputs by two scripts in \texttt{code/}; \SPIDtestsN\ automated checks (\texttt{tests/}) verify that the exports reconcile with each other, that regeneration is byte-identical, and that the generator contains no data literal. The exports themselves were regenerated read-only on the master node in this revision and found byte-identical to the committed ones except for the ordering of tied rows in two files.19\end{itemize}2021\subsection{Mechanical accuracy audits (implemented in this revision)}\label{sec:val_mechanical}2223\paragraph{Field-level grounding.} The prompt requires that numbers and locations appear in the text, and the structured fields are literal tokens, so their containment in the stored section text can be tested for every mention. For all \SPIDmentions\ mentions joined to their section text (\texttt{results/grounding/}; \SPIDgroundBootstrapReps\ bootstrap replicates clustered by section), every listed year is found in \SPIDgroundYearsPct\% of mentions with years (\SPIDgroundYearsCiLow--\SPIDgroundYearsCiHigh), every listed partner in \SPIDgroundPartnersPct\%, the amount as a number (in any of the usual renderings) in \SPIDgroundAmountNumberPct\% and with a currency token within forty characters in \SPIDgroundAmountCurrencyPct\%, every listed location in \SPIDgroundLocationsPct\% (10-K \SPIDgroundLocationsTenK\%, 10-Q \SPIDgroundLocationsTenQ\%, 8-K \SPIDgroundLocationsEightK\%), and every technology tag in \SPIDgroundTechPct\%; at the level of individual elements the rates are \SPIDgroundElemYears\% for years, \SPIDgroundElemPartners\% for partners, \SPIDgroundElemLocations\% for locations and \SPIDgroundElemTech\% for technologies. All structured fields of a mention are found in \SPIDgroundAllPct\% of the \SPIDgroundAllN\ mentions that carry at least one (10-K \SPIDgroundAllTenK\%, 10-Q \SPIDgroundAllTenQ\%, 8-K \SPIDgroundAllEightK\%; June block \SPIDgroundAllJune\%, October \SPIDgroundAllOctober\%), with no difference between filings before and after the model's training cut-off beyond one percentage point. Three readings of these rates are needed. First, they are rates of \emph{transcription}, not of correctness: a year, partner or amount can be copied faithfully from a passage that describes a counterparty's initiative, a hypothetical, or a financing instrument, so grounding is necessary for a correct mention and not sufficient. Second, the lower rates for locations and technologies reflect normalization by the model (``Texas'' for ``Austin, TX''; ``cloud computing'' for ``AWS'') as much as invention; the element-level rate for locations is \SPIDgroundElemLocations\%. Third, \SPIDyearsAbsentFromText\ mentions carry a year absent from the section, and the technology and year lists equal the regex output on the model's own description for up to \SPIDtechRegexEqualPct\% and \SPIDyearsRegexEqualPct\% of mentions, so for those fields ``grounded'' may mean ``copied from the paraphrase'' (\Cref{sec:contract}). The breakdown by field and form is in \Cref{tab:grounding} and \Cref{fig:grounding}; the full breakdown by item is in \texttt{results/grounding/grounding\_by\_field\_form\_item.csv}.2425\input{tables/tab_grounding}2627\input{figures/fig_grounding}2829\paragraph{What the model was shown.} \Cref{sec:focus} reports the shown fraction of every ledgered section. The recall of the pipeline on long sections is bounded from above by what the windows contain: on the over-budget sections the rule-based extractor of the code base recovers \SPIDrulesFocusRecoveredPct\% of its full-text mentions from the focused text (\SPIDrulesFocusMdnaPct\% in the 10-K MD\&A). Gate~(e) is therefore reported as met at passage level, not at section level.3031\paragraph{External consistency with 8-K item codes.} Form 8-K bodies name the items they report under, and \SPIDeightKItemCodePct\% of the \SPIDledgerSectionsEightK\ 8-K sections carry at least one item code in their text, which gives an external classification at no cost (\texttt{results/benchmark\_8k/}). Among 8-Ks filed under Item~2.01 (completion of an acquisition or disposition; \SPIDbenchTwoOhOneN\ sections) the model returned at least one \texttt{ma\_integration} mention for \SPIDbenchMaGivenTwoOhOne\ (\SPIDbenchMaGivenTwoOhOneCiLow--\SPIDbenchMaGivenTwoOhOneCiHigh) of sections, against \SPIDbenchMaGivenEightOhOne\ under Item~8.01 (other events) and \SPIDbenchMaGivenFiveOhTwo\ under Item~5.02 (officer changes) and a base rate of \SPIDbenchMaBase; conversely \SPIDbenchItemsGivenMa\ of 8-K \texttt{ma\_integration} mentions sit in a filing under Item~2.01 or 1.01. Under Item~2.05 (exit or disposal costs; \SPIDbenchTwoOhFiveN\ sections) the model returned a \texttt{cost\_reduction} mention for \SPIDbenchCostGivenTwoOhFive\ (\SPIDbenchCostGivenTwoOhFiveCiLow--\SPIDbenchCostGivenTwoOhFiveCiHigh) of sections, a lift of about \SPIDbenchCostLift\ over the base rate, and \SPIDbenchTwoOhFiveGivenCost\ of \texttt{cost\_reduction} mentions sit under that item; Item~1.01 (material agreements) carries a \texttt{partnership} mention in \SPIDbenchPartnershipGivenOneOhOne\ of cases against a base rate of \SPIDbenchPartnershipBase. These are consistency rates with an administrative classification of the \emph{filing}, not precision or recall of the \emph{mention}: an Item~2.01 filing without an M\&A mention may describe a disposition, and an Item~8.01 filing with one may be a genuine acquisition announced under a general item. They show that the dominant category is not an artefact of the extractor's vocabulary; they do not measure its accuracy (\Cref{tab:benchmark_8k}).3233\input{tables/tab_benchmark_8k}3435\paragraph{Agreement with a rule-based baseline.} The code base contains a rule-based extractor (\texttt{rules.py}) that needs no model call. Run on a seeded, stratified sample of \SPIDrulesSampleProductive\ productive and \SPIDrulesSampleZero\ zero-mention ledger sections with inverse-probability weights (\texttt{results/rules/}), the rules flag at least one candidate in \SPIDrulesGivenLlmPct\% of the sections where the model returned a mention (10-K \SPIDrulesGivenLlmTenK\%, 10-Q \SPIDrulesGivenLlmTenQ\%, 8-K \SPIDrulesGivenLlmEightK\%; June \SPIDrulesGivenLlmJune\%, October \SPIDrulesGivenLlmOctober\%), while the model returns a mention in \SPIDllmGivenRulesPct\% of the sections the rules flag; the rules flag \SPIDrulesFlagPct\% of ledgered sections against \SPIDproductiveShareLedger\% for the model, section-level agreement is \SPIDrulesAgreementPct\% (model-only \SPIDrulesLlmOnlyPct\%, rules-only \SPIDrulesOnlyPct\%), and on co-detected sections the model's category is among the rule categories in \SPIDrulesTypeOverlapPct\% of cases (\texttt{ma\_integration} \SPIDrulesTypeMa\%, \texttt{drug\_pipeline} \SPIDrulesTypeDrug\%, \texttt{cost\_reduction} \SPIDrulesTypeCost\%, \texttt{capex\_program} \SPIDrulesTypeCapex\%, \texttt{plant\_construction} \SPIDrulesTypePlant\%, \texttt{geographic\_expansion} \SPIDrulesTypeGeo\%). Neither extractor is a ground truth, so this is a measure of the marginal information in the model step---it abstains on most of what the lexical rules flag and rarely fires where they do not---rather than of accuracy. A full-corpus run of the rules is deferred; the sample gives roughly half a percentage point of precision on the rates (\Cref{tab:rules_agreement}).3637\input{tables/tab_rules_agreement}3839\paragraph{Known false-positive classes.} The financing-instrument pattern of \Cref{sec:construct_def} matches \SPIDfinMentionsPct\% of mentions, concentrated in Part~II Item~2 of the 10-Q and in 8-K bodies, and \Cref{sec:results} restates the category counts without them. A related parser check finds that the 10-Q Part~II Item~2 section (median \SPIDunregMedianWords\ words) is longer than \SPIDunregLongThreshold\ words in \SPIDunregGtThreeK\ of \SPIDitemTwoPairs\ filings (\SPIDunregGtThreeKPct\%), \SPIDswapCandidates\ of which have an MD\&A shorter than 500 words and are candidates for a Part~I/Part~II ``Item~2'' swap by the segmenter; \SPIDunregGtThreeKMentions\ mentions come from those sections. Both classes are detected by pattern, not by annotation, so their sizes are indicative, and the annotation protocol labels them explicitly.4041\paragraph{Resolver sensitivity.} The dependence of the project count on the key and on the confidence threshold is reported in \Cref{sec:speccurve}; the over-merge indicators (distinct names and partners per project, amounts taken from a different counterparty) are in \Cref{sec:resolution_rule}.424344\subsection{Accuracy evaluation against human labels (frame drawn, labels not produced)}\label{sec:val_human}4546\paragraph{Sampling frame.} The unit is the ledgered section, that is, exactly the text block the extractor received a verdict for. The frame was drawn with a fixed seed (\SPIDannSeed) from the two populations of the ledger (\texttt{results/annotation\_frame/}; protocol in \texttt{docs/annotation\_protocol.md}). From the \SPIDannPopProductive\ productive sections, \SPIDannStrata\ strata are defined by item group, run (June block or October), the shown-fraction bin of \Cref{sec:focus}, the confidence bin of the section's best mention and the category group; allocation is proportional with a floor of \SPIDannFloor\ per stratum and a \SPIDannOversample-fold oversampling of the confidence bin at or below 0.7, which gives $n = \SPIDannNProductive$ (the floor dominates; a nested core subsample of \SPIDannCore\ drawn proportionally within the frame is flagged for a first pass). From the \SPIDannPopZero\ zero-mention sections, \SPIDannNZero\ are drawn with inclusion probability proportional to the gate verdict and to the logarithm of section length, so that gate-rejected sections are represented (\SPIDannZeroGatePass\ gate-passing, \SPIDannZeroGateReject\ gate-rejected) and false negatives can be estimated with inverse-probability weights; the earlier version's equal-count draw without weights would have biased recall upward. The frame is split into a development half (\SPIDannDev\ sections), on which prompt and threshold changes may be tuned, and a locked holdout half (\SPIDannHoldout), used once per release. The design effects are \SPIDannDeffProductive\ (productive) and \SPIDannDeffZero\ (zero-mention); the 95\% half-widths are $\pm$\SPIDannHalfWidthPrecision\ percentage points for a precision near 0.5 and $\pm$\SPIDannHalfWidthFn\ points for a false-negative rate near 5\%, and the minimum detectable difference in the false-negative rate between gate-passing and gate-rejected strata is \SPIDannMde\ points. The full-text frames, the text the model was shown, and the sealed model output are stored on the master node (\SPIDannFrameMb~MB); the worktree copies carry the text truncated to \SPIDannTextChars\ characters.4748\paragraph{Labels and annotators.} Two annotators will independently code each section without seeing the model output (it is unsealed only for the mention-level comparison): at the section level, whether the section describes at least one project and how many (labels S1--S2); at the mention level after unsealing, for each initiative its presence, category (from the \SPIDtaxonomyN-code taxonomy), status, cited years, location, partners, amount and amount type, plus the six-way label of \Cref{sec:construct_def} (project, generic strategy, third-party, financing instrument, uncertain, none) and a boilerplate flag (M1--M4); and at the project level, whether a cluster is correct, over-merged or under-merged (L1). Disagreements will be adjudicated by the third team member; inter-annotator agreement will be reported as Krippendorff's $\alpha$ and prevalence-adjusted $\kappa$ for presence and category, and as exact-match rates for fields. The annotators are the authors; no label has been produced at the time of writing, and no label generated by a language model will be reported as a human label.4950\paragraph{Metrics.} On the holdout, with the design weights: candidate-gate recall (share of annotated projects whose section passed the gate); project-detection precision, recall and F1 at the mention level, by form, item and shown-fraction bin; category macro- and micro-F1; field accuracy for amount, years and location; the abstention rate (sections where the model returned an empty list) conditional on annotated content; the share of mentions in each false-positive class; and confidence calibration (precision by confidence bin, with bootstrap intervals, the 0.6 bin reported with and without the defaulted values). The same metrics will be computed for the rule-based extractor, so that the marginal value of the model step is documented, and the estimated error structure will be the input to the measurement-error correction that \citet{ludwig2025} describe.5152\paragraph{Linkage audit.} From the resolved projects, a sample stratified by number of mentions (1, 2--5, 6--10, more than 10) and by category will be reviewed to label each project cluster as correct, over-merged (contains distinct initiatives) or under-merged (a duplicate of another cluster of the same issuer). Link precision and recall and the error costs of false merges versus missed links will be reported, with particular attention to long-running programs, subsidiaries, near-duplicate names and the generic-token keys of \Cref{sec:resolution_rule}.5354\paragraph{Stability of the instrument.} A protocol of \SPIDstabilitySections\ sections re-submitted three times, once with a paraphrased prompt and once to a substitute model (\SPIDstabilityCalls\ calls), and a re-submission of the \SPIDresumeBatchCalls\ gate-passing sections of the suspect post-outage batch, are specified; both require new model calls, which were not authorised for this revision, and their results are absent from this paper.5556\paragraph{Temporal test and missingness.} For a sample of projects with at least three mentions, the status and amount reconstructed using only mentions filed on or before each date will be compared with the full-information values, to verify that no later information leaks into earlier states and to document how often status reverses. Zero-project issuer-years will be cross-tabulated by sector, size and filing count to distinguish ``no disclosure'' from ``not processed''; the former is informative, the latter is a data gap, and the \texttt{coverage\_status} field (complete for the defined scope, partial, failed, not observed) proposed for the issuer-year export will carry that distinction.5758\subsection{Acceptance gates}5960\begin{table}[H]61\centering\small62\caption{Acceptance gates for a research-ready release and their status at revision \texttt{2324b58} after the Stage D audits}\label{tab:gates}63\begin{threeparttable}64\begin{tabularx}{\textwidth}{@{}l>{\raggedright\arraybackslash}p{5.6cm}>{\raggedright\arraybackslash}X@{}}65\toprule66Gate & Requirement & Status \\67\midrule68(a) & Target issuer universe and time window frozen and auditable & Partially met: snapshot design; mapped universe and derived inputs archived with SHA-256 checksums; the source holdings file and SEC ticker file are not archived and the lines dropped before mapping are not listed \\69(b) & Every published project links to at least one verifiable primary-source passage & Met at section level; structured fields mechanically checked against the section text (\SPIDgroundAllPct\% fully grounded); description fidelity not checkable; character spans not stored \\70(c) & Duplicate, temporal and amount rules documented and tested & Documented; the pipeline snapshot contains no test; the exhibit layer has \SPIDtestsN\ automated checks that do not test the pipeline; no event or amount-type tables; representative fields depend on tie order \\71(d) & Extraction and matching accuracy evaluated on independent human labels & Not met: protocol revised, frame drawn and weighted ($n = \SPIDannNProductive + \SPIDannNZero$), annotation pending (author task) \\72(e) & Issuer/form/year coverage and failures quantified & Met at passage level: coverage, parse and scope failures, gate rejections, outage, silent failures and shown fractions quantified; one post-outage batch flagged as suspect \\73(f) & Every empirical result regenerable from a pinned build & Exhibits and exports regenerable from the released code and data (\texttt{run\_all.sh}, lock files); the databases are regenerable by the authors on the master node; the extraction step is re-runnable and ledgered, not deterministic; release of the derived tables is an author decision not yet taken \\74(g) & All coauthors have reviewed scientific claims and authorship & Pending \\75\bottomrule76\end{tabularx}77\begin{tablenotes}\footnotesize78\item Gates are those of the research-design specification (\Cref{sec:sample}). ``Met at passage level'' means that the quantity is established for the text the model was actually shown, not for the whole section. Items marked as author tasks or decisions are listed in \Cref{sec:limitations}.79\end{tablenotes}80\end{threeparttable}81\end{table}8283Until gates (c), (d) and (g) are met, the database is a \emph{research database under validation}, and any use of its counts or amounts in empirical work should treat extraction and linkage error as measurement error of unknown structure, with the audits above as the only available bounds.84