spb/firc0001_uqo

1% ============================================================================2\section{Validation Framework and Current Audit}\label{sec:validation}3% ============================================================================45The aim of identifying ``all projects'' of an issuer is a \emph{coverage target}, not an assumption. Only projects that appear in the captured sources, survive parsing and gating, are recognized by the model and are correctly linked are observable, and each of these steps loses or distorts some. This section specifies how those losses are to be measured and reports what has been measured so far. The headline is simple: the \emph{coverage} and \emph{integrity} components are implemented and quantified; the \emph{accuracy} components---which require independent human labels---are specified but not yet executed. Table~\ref{tab:gates} summarizes the status of the acceptance gates.67\subsection{Coverage and integrity checks (implemented)}89\begin{itemize}10\item \textbf{Universe reconciliation} (Table~\ref{tab:reconciliation}): every input line of the Russell proxy is accounted for as matched, dropped-as-non-company, or unmatched; the 65 issuers without filings are listed by name and reason.11\item \textbf{Acquisition completeness:} for every issuer the SEC submissions index is enumerated in full (recent block and archive blocks) and every retained filing is logged; download failures are counted per issuer and were zero in the final run. Filing counts by form and year are published (Table~\ref{tab:coverage_year}).12\item \textbf{Parsing coverage:} 16 of 527{,}548 primary documents could not be parsed (Table~\ref{tab:filings}); filings without recognizable Item headings are kept as a single section rather than dropped.13\item \textbf{Extraction ledger:} every section submitted to the model is recorded; the ledger was audited after the credit-exhaustion incident and all sections processed during the outage were re-submitted. The ledger does not yet record a per-call status, so an undetected API failure would still appear as ``no project''; the run logs (retried errors, JSON parse failures: five in the June run, one in the October run) are the only current evidence that such failures were rare.14\item \textbf{Referential integrity:} foreign keys from chunks to sections to filings to companies are enforced by the database; the merge of ten shards is append-only on globally unique keys and was checked for duplicate accession numbers and section identifiers (none).15\item \textbf{Reproducibility of exhibits:} every table and figure of this paper is regenerated from the databases by two scripts in \texttt{code/}; the numbers can be compared against \texttt{data/stats.json}.16\end{itemize}1718\subsection{Accuracy evaluation (specified, not executed)}1920\paragraph{Gold standard.} A stratified sample of sections will be drawn from the ledger with strata defined by form (10-K, 10-Q, 8-K), item, issuer size (fund weight tercile), sector, filing year (2010--2014, 2015--2019, 2020--2026) and disclosure length. The sample will include sections that produced mentions and, in equal number, randomly selected sections that produced none, so that both false positives and false negatives can be estimated. Two annotators will independently code each section: for each initiative present, its name, category (from the 21-code taxonomy), status, cited years, location, partners, amount and amount type, plus the five-way label of Section~\ref{sec:construct} (project, generic strategy, third-party, uncertain, none). Disagreements will be adjudicated by the third team member; inter-annotator agreement (Cohen's $\kappa$ for presence and category, exact-match rates for fields) will be reported. The sample will be split into a development half, on which prompt and threshold changes may be tuned, and a locked holdout half, used once per release.2122\paragraph{Metrics.} On the holdout: candidate-gate recall (share of annotated projects whose section passed the gate); project-detection precision, recall and F1 at the mention level; category macro- and micro-F1; field accuracy for amount, years and location; the abstention rate (sections where the model returned an empty list) conditional on annotated content; and confidence calibration (precision by confidence bin, with bootstrap intervals). The same metrics will be computed for the rule-based extractor and, if budget allows, for a semantic-retrieval baseline, so that the marginal value of the LLM step is documented.2324\paragraph{Linkage audit.} From the resolved projects, a sample stratified by number of mentions (1, 2--5, 6--10, more than 10) and by category will be reviewed to label each project cluster as correct, over-merged (contains distinct initiatives) or under-merged (a duplicate of another cluster of the same issuer). Link precision and recall and the error costs of false merges versus missed links will be reported, with particular attention to long-running programs, subsidiaries and near-duplicate names.2526\paragraph{Temporal test.} For a sample of projects with at least three mentions, the status and amount reconstructed using only mentions filed on or before each date will be compared with the full-information values, to verify that no later information leaks into earlier states and to document how often status reverses.2728\paragraph{Missingness.} Zero-project issuer-years will be cross-tabulated by sector, size and filing count to distinguish ``no disclosure'' from ``not processed''; the former is informative, the latter is a data gap, and the \texttt{coverage\_status} field (complete for the defined scope, partial, failed, not observed) proposed for the issuer-year export will carry that distinction.2930\subsection{Acceptance gates}3132\begin{table}[H]33\centering\small34\caption{Acceptance gates for a research-ready release and their status at revision \texttt{2324b58}}\label{tab:gates}35\begin{tabularx}{\textwidth}{lXl}36\toprule37Gate & Requirement & Status \\38\midrule39(a) & Target issuer universe and time window frozen and auditable & Met (snapshot design; file hashes pending) \\40(b) & Every published project links to at least one verifiable primary-source passage & Met at section level; character spans not stored \\41(c) & Duplicate, temporal and amount rules documented and tested & Documented; unit tests partial; no event or amount-type tables \\42(d) & Extraction and matching accuracy evaluated on independent human labels & Not met (protocol specified) \\43(e) & Issuer/form/year coverage and failures quantified & Met \\44(f) & Every empirical result regenerable from a pinned build & Met for this paper (\texttt{code/}, revision \texttt{2324b58}) \\45(g) & All coauthors have reviewed scientific claims and authorship & Pending \\46\bottomrule47\end{tabularx}48\end{table}4950Until gates (c), (d) and (g) are met, the database is a \emph{research database under validation}, and any use of its counts or amounts in empirical work should treat extraction and linkage error as measurement error of unknown magnitude.51