Validation report — FIRC-0001, Stage D node batches N0–N13 (M3U96b, read-only)
Executor: node batch (Agent "N"), 2026-11-05, 14:08–15:05 local. Worktree UQO-WP-REVIEW/worktrees/FIRC-0001, branch review-20261103. Protocol: Part IX (peer_reviews/_briefs/protocol_part_ix_reproducibility.md). Companion: validation_report.md (laptop executor, exhibit layer).
1. What was re-run, where, how long
| package | script (code/audit/) |
node | wall | reads | writes (worktree) |
|---|---|---|---|---|---|
| N0 | a00_probe_schema.py, common.py |
M3U96b | 8 s | both DuckDB files (schema) | results/n0_probe/, spid_snapshot/SHA256SUMS |
| N1 | a01_ledger_reconciliation.py |
M3U96b | 12 s | extracted_sections, project_mentions, parquet text (RE2 gate scan 29 GB), both run logs |
results/ledger/ (18 files) |
| N3 | a03_resolver_speccurve.py |
M3U96b | 2 s | project_mentions, projects, src_companies |
results/resolver/ (14 files) |
| N4 | a04_git_evidence.sh, a04_runs_segmenters.py |
M3U96b | 32 s | git metadata (read-only), spid_sections + parquet text of 7,368 filings, ledger, mentions |
results/runs/ (19 files) |
| N2 | a02_grounding.py |
M3U96b | 217 s | text of the 168,230 productive sections (6.0 GB) | results/grounding/ (13 files; mention_flags.parquet node-only) |
| N5 | a05_focus_coverage.py |
M3U96b | 585 s | text of all 1,123,098 ledger sections (23.4 GB) | results/focus/ (9 files; section_features.parquet node-only) |
| N6 | a06_false_positive_classes.py |
M3U96b | < 10 s | mentions, projects, 10-Q section word counts | results/fp_classes/ (10 files) |
| N9 | a09_timeseries_cohort.py |
M3U96b | < 10 s | mentions, src_filings |
results/timeseries/ (4 files) |
| N10 | a10_universe_artefacts.py |
M3U96b | 1 s | data/metadata/*.json, edgar.filings, annual_panel |
results/universe/, data/metadata/ (3 JSON + SHA256SUMS) |
| N7 | a07_8k_item_benchmark.py |
M3U96b | 8 s | 8-K text (item-code regex), mentions | results/benchmark_8k/ (5 files) |
| N8 | a08_rules_baseline.py |
M3U96b | see §4 | stratified seeded sample of 60,000 ledger sections (text) | results/rules/ (rules_mentions.parquet node-only) |
| N11 | a11_annotation_frame.py |
M3U96b | 9 s | N5 features, N8 sample, ledger, mentions, text of 2,062 sections | results/annotation_frame/ (7 files), node frames/ |
| N12 | a12_env_lock.sh |
M3U96b | 2 s | venv metadata | requirements.lock, .python-version, environment.md, results/env/ |
| N13 | a13_export_stats_check.sh + L1's code/export_stats.py |
M3U96b | 44 s | both DuckDB files | results/export_check/, data/stats.json + 4 new CSV |
Settings everywhere: duckdb.connect(read_only=True), SET threads=4, SET memory_limit='24GB' (16 GB inside export_stats.py), SET temp_directory='~/firc0001_review/tmp', nice -n 10, OMP_NUM_THREADS=4, Python 3.12.15 / duckdb 2.0.0 (environment.md), seed 20261105. Zero LLM/API calls; no network beyond ssh. Logs: reproducibility/FIRC-0001/logs/node/. Integrity of the inputs: spid.duckdb 3,155,701,760 B mtime 2026-10-28 12:21:11 and edgar_research.duckdb 73,500,995,584 B mtime 2026-10-27 21:34:46 before and after (every manifest.json); ~/UQO_RESEARCH_DB git d700868, 0 changed paths. One incident: the first schema probe (before temp_directory was set) ran out of memory on a text semi-join and DuckDB created then removed its default spill directory spid.duckdb.tmp next to the database (parent-directory mtime 14:09); the database file is unchanged (size, mtime) and no data was written.
2. What reconciles (BEFORE = paper / committed data; AFTER = measured)
| quantity | paper / data/ |
measured | verdict |
|---|---|---|---|
| ledger rows | 1,123,098 | 1,123,098 | = |
| stored mentions / projects | 308,065 / 121,099 | 308,065 / 121,099; _keyed rebuilt from resolve.py gives the identical project-id set |
= |
| ledger mentions (Table 3) | 308,477 | 308,477 = 308,065 + 412 over 304 sections (10-K 127 / 10-Q 257 / 8-K 28) | = (explained) |
| June block | 153,894 sections / 39,930 mentions / 19,227 projects | 154,964 rows at one timestamp = 153,894 + 1,070 rows < 30 words; 39,930 mentions; 19,227 projects when resolved alone | = (gap explained) |
| S&P 500 projects after October | 27,766 | 27,766 (15,910 June-only, 3,317 both, 8,539 October-only) | = |
| June 10-Qs | 3,710 (RUSSELL3000-EXTENSION.md) |
3,710 accessions in the backfilled block | = |
| outage rows removed | 149,352 (research log) | 1,117,479 completed − 968,127 pending = 149,352; 0 ledger rows in the 5.25 h gap | = |
| median 10-K MD&A words | 9,905 | 9,905 (n = 28,919; 64,194 characters) | = |
| 10-K ledger funnel | 143,704 | 341,207 − 188,518 − 35,037 − 1,112 = 116,540 + 27,164 legacy = 143,704 | = |
| universe | 2,573 → 2,555 → 2,564; 7 unmatched | 2,573 lines − 18 dual-class CIKs = 2,555; 500 + 2,555 − 491 = 2,564; 7 unmatched | = |
| first-seen range 2011–2025 | "6,200–8,200" | 6,240–8,216 | = |
confidence = 0.600 |
(Agent 8 probe) 3,278 | 3,278 | = |
gate (DuckDB RE2 vs Python re) |
— | 703,737 = 703,737; 0 productive sections fail | = |
_focus replica vs real function |
— | 0 mismatches on 1,123,098 sections | = |
| 28 baseline CSV exports | committed 678d233 | 26 byte-identical; projects_by_index.csv, top_issuers.csv identical as sets, rows reordered among ties (no deterministic ORDER BY in the SQL) |
= (order) |
3. What changed (the numbers the manuscript must adopt; details in implemented_changes.md, "N-batches")
- "Sections submitted" → true calls: 703,725 of 1,123,098 (419,361 rows never called); productive share 23.9 % of calls (15.0 % of rows); 0.44 mentions per call; ≈ 0.0013 USD per call.
- Silent failures: 14 JSON-decode failures outside the outage (paper: "one"), June 3; and one post-resume batch of 5,198 sections (2,686 calls) with 23 productive vs 536 expected and no logged failure — suspect block, settable only by re-submission (AR-LLM-4 proposed in
unresolved_issues.md). - Throughput: 13.68 h wall-clock, 5.25 h gap, 31.9 rows/s = 20.1 calls/s effective (paper: "about twelve hours, 27–36/s").
- Project count: 121,099 under the current key; range 96,749 (type dropped) – 198,488 (strict full-name + location), 21,027 under cik|type, 308,065 at mention level; 54.8 % of projects keyed on a name token,
acquisitionalone absorbing 40,407 mentions; 33,820 projects have arg_max ties (394 names / 600 descriptions differ between the authors' run and ours). - Grounding (new): years 99.8 %, partners 99.4 %, amount number 93.7 % (80.5 % with currency within 40 chars), locations 78.3 %, technology tags 63.6 %; all structured fields 88.2 % (8-K 76.6 %); regex-backfill upper bounds 39 % (years) / 46 % (technologies).
- Focus coverage (new): median shown fraction 9.3 % of a 10-K MD&A, last window at 16 %; 62.7 % of mentions from sections shown < 25 %; mention yield plateaus at ≈ 2.5 per 10,000 shown characters; 0 messages cut at 14,000; only 169 (0.6 %) MD&As < 30 words (the "2,081" is the legacy substitution).
- Two runs / two segmenters: no June git revision; extractor files dated 2026-06-17; legacy MD&A text has half the money tokens of the parsed text; 86
full_document10-Ks (35 invisible); cohort effect +8.8 → −23.0 projects per issuer when the run is held fixed;fiscal_yearNULL for 68,827 mentions with no stored source to fill it. - Construct violations: 7.3 % of mentions match the financing/buyback class (54.5 % of 10-Q unregistered_sales, 52.6 % of capex_program).
- Amounts: max 10¹² (1 row), 161 mentions ≥ 50 B (104 projects), 584 zeros with
amount_rawNULL, 24 negatives;amount_rawis a formatted float with 584 deviations (not 1). - 8-K proxy (new): P(M&A mention | Item 2.01) = 0.91 vs 0.03 (Item 5.02); P(cost_reduction | Item 2.05) = 0.71.
- Fixed cohort: new projects per issuer 4.7 → 2.8 (2011 → 2025) with mentions per issuer flat (9.6 → 8.8).
- Universe: holdings CSV and
company_tickers.jsonnot archived (2,587 / 14 / 21 not reconstructible);annual_panelhas 2,499 CIKs (paper 2,501); 2 filings missing on disk (XEL 10-K 2014, LNT 10-Q 2012), 14 parse errors.
4. Rules baseline and annotation frame (N8, N11)
N8 (200 s, stratified seeded sample 30,000 + 30,000 sections; a first run had drawn a simple random sample because DuckDB applies USING SAMPLE before WHERE — corrected with subqueries and re-run): P(rules ≥ 1 | LLM ≥ 1) = 85.7 % (10-K 97.4, 10-Q 94.4, 8-K 69.2), weighted P(LLM ≥ 1 | rules ≥ 1) = 32.5 %, agreement 71.2 %, type overlap on co-detected sections 84.6 % (M&A 96.8 %); rules on _focus(text) recover 26.2 % of the rules-on-full-text mentions in over-budget sections (10-K MD&A 20.8 %). N11 (9 s): 1,677 productive sections in 192 strata (+ core 510) and 385 zero-mention sections, weights and dev/holdout split, ± 4.4 pp / ± 2.4 pp design precision; frames with full text on the node, protocol in docs/annotation_protocol.md; no label produced.
5. Export regeneration (N13) and reproducibility verdict
code/export_stats.py (L1 revision 64ddafa, sha256 a7bf4939…) re-run on M3U96b in 44 s into a scratch directory: 26 of the 28 baseline CSVs byte-identical, 2 identical up to the order of tied rows, stats.json 17 → 41 keys (example_project removed as intended), 4 new files (calls_by_form.csv, cohort_by_run.csv, ledger_collisions.csv, projects_by_type_ci.csv). The regenerated stats.json and the 4 new files are copied into data/; the 2 reordered files are left as committed (content identical). Cross-checks against the N audits: gate_pass_total 703,737 = N1; calls_by_form = N1; backfilled_rows 154,964 / _under_30_words 1,070 = N1; outage 01:25:35 → 06:40:45, 5.25 h = N1; conf_exact_default_count 3,278 = N1; ledger_10k_mdna_median_words 9,905 = N5; annual_panel_firms 2,499 = N10; amount_ge_50b 104 projects = N2. Four values of the L1 script disagree with the audits and must be corrected by its owner before Stage E uses them: amount_raw_deviations = 0 (the <> comparison is NULL-blind; IS DISTINCT FROM gives 584), october_wall_clock_hours = 22.05 and october_effective_sections_per_second = 16.0 (measured from the backfill timestamp 13:12 instead of the first October flush 21:35; correct 13.68 h and 31.9 rows/s), spid_runs.sp500_run.sections_logged = 4,271 (first batch, not 153,894). Deterministic ordering (ORDER BY … , cik / , ticker) is recommended for projects_by_index and top_issuers.
Verdict for the data layer: R1 — the exports are reproduced from the documented analysis data (the two DuckDB files on the master node; raw filings 254.5 GB not needed), with the two order-only differences and the four flagged stats.json values listed above. The audit numbers of N1–N11 are new measurements, each produced by a committed script with a manifest; none changes a conclusion of the paper's thesis ("research database under validation, gate (d) not met"); several weaken positive statements as Rule 2 requires (R4 not triggered).