spb/firc0001_uqo

1# Validation report — FIRC-0001, Stage D node batches N0–N13 (M3U96b, read-only)23Executor: node batch (Agent "N"), 2026-11-05, 14:08–15:05 local. Worktree `UQO-WP-REVIEW/worktrees/FIRC-0001`, branch `review-20261103`. Protocol: Part IX (`peer_reviews/_briefs/protocol_part_ix_reproducibility.md`). Companion: `validation_report.md` (laptop executor, exhibit layer).45## 1. What was re-run, where, how long67| package | script (`code/audit/`) | node | wall | reads | writes (worktree) |8|---|---|---|---|---|---|9| N0 | `a00_probe_schema.py`, `common.py` | M3U96b | 8 s | both DuckDB files (schema) | `results/n0_probe/`, `spid_snapshot/SHA256SUMS` |10| N1 | `a01_ledger_reconciliation.py` | M3U96b | 12 s | `extracted_sections`, `project_mentions`, parquet text (RE2 gate scan 29 GB), both run logs | `results/ledger/` (18 files) |11| N3 | `a03_resolver_speccurve.py` | M3U96b | 2 s | `project_mentions`, `projects`, `src_companies` | `results/resolver/` (14 files) |12| N4 | `a04_git_evidence.sh`, `a04_runs_segmenters.py` | M3U96b | 32 s | git metadata (read-only), `spid_sections` + parquet text of 7,368 filings, ledger, mentions | `results/runs/` (19 files) |13| N2 | `a02_grounding.py` | M3U96b | 217 s | text of the 168,230 productive sections (6.0 GB) | `results/grounding/` (13 files; `mention_flags.parquet` node-only) |14| N5 | `a05_focus_coverage.py` | M3U96b | 585 s | text of all 1,123,098 ledger sections (23.4 GB) | `results/focus/` (9 files; `section_features.parquet` node-only) |15| N6 | `a06_false_positive_classes.py` | M3U96b | < 10 s | mentions, projects, 10-Q section word counts | `results/fp_classes/` (10 files) |16| N9 | `a09_timeseries_cohort.py` | M3U96b | < 10 s | mentions, `src_filings` | `results/timeseries/` (4 files) |17| N10 | `a10_universe_artefacts.py` | M3U96b | 1 s | `data/metadata/*.json`, `edgar.filings`, `annual_panel` | `results/universe/`, `data/metadata/` (3 JSON + SHA256SUMS) |18| N7 | `a07_8k_item_benchmark.py` | M3U96b | 8 s | 8-K text (item-code regex), mentions | `results/benchmark_8k/` (5 files) |19| N8 | `a08_rules_baseline.py` | M3U96b | see §4 | stratified seeded sample of 60,000 ledger sections (text) | `results/rules/` (`rules_mentions.parquet` node-only) |20| N11 | `a11_annotation_frame.py` | M3U96b | 9 s | N5 features, N8 sample, ledger, mentions, text of 2,062 sections | `results/annotation_frame/` (7 files), node `frames/` |21| N12 | `a12_env_lock.sh` | M3U96b | 2 s | venv metadata | `requirements.lock`, `.python-version`, `environment.md`, `results/env/` |22| N13 | `a13_export_stats_check.sh` + L1's `code/export_stats.py` | M3U96b | 44 s | both DuckDB files | `results/export_check/`, `data/stats.json` + 4 new CSV |2324Settings everywhere: `duckdb.connect(read_only=True)`, `SET threads=4`, `SET memory_limit='24GB'` (16 GB inside `export_stats.py`), `SET temp_directory='~/firc0001_review/tmp'`, `nice -n 10`, `OMP_NUM_THREADS=4`, Python 3.12.15 / duckdb 2.0.0 (`environment.md`), seed 20261105. Zero LLM/API calls; no network beyond ssh. Logs: `reproducibility/FIRC-0001/logs/node/`. Integrity of the inputs: `spid.duckdb` 3,155,701,760 B mtime 2026-10-28 12:21:11 and `edgar_research.duckdb` 73,500,995,584 B mtime 2026-10-27 21:34:46 before and after (every `manifest.json`); `~/UQO_RESEARCH_DB` git `d700868`, 0 changed paths. **One incident:** the first schema probe (before `temp_directory` was set) ran out of memory on a text semi-join and DuckDB created then removed its default spill directory `spid.duckdb.tmp` next to the database (parent-directory mtime 14:09); the database file is unchanged (size, mtime) and no data was written. The same transient spill directory was created and removed by L1's `export_stats.py` during N13 (its connections set no `temp_directory`; parent-directory mtime 14:43), again with the database file byte-unchanged.2526## 2. What reconciles (BEFORE = paper / committed data; AFTER = measured)2728| quantity | paper / `data/` | measured | verdict |29|---|---|---|---|30| ledger rows | 1,123,098 | 1,123,098 | = |31| stored mentions / projects | 308,065 / 121,099 | 308,065 / 121,099; `_keyed` rebuilt from `resolve.py` gives the identical project-id set | = |32| ledger mentions (Table 3) | 308,477 | 308,477 = 308,065 + 412 over 304 sections (10-K 127 / 10-Q 257 / 8-K 28) | = (explained) |33| June block | 153,894 sections / 39,930 mentions / 19,227 projects | 154,964 rows at one timestamp = 153,894 + 1,070 rows < 30 words; 39,930 mentions; 19,227 projects when resolved alone | = (gap explained) |34| S&P 500 projects after October | 27,766 | 27,766 (15,910 June-only, 3,317 both, 8,539 October-only) | = |35| June 10-Qs | 3,710 (`RUSSELL3000-EXTENSION.md`) | 3,710 accessions in the backfilled block | = |36| outage rows removed | 149,352 (research log) | 1,117,479 completed − 968,127 pending = 149,352; 0 ledger rows in the 5.25 h gap | = |37| median 10-K MD&A words | 9,905 | 9,905 (n = 28,919; 64,194 characters) | = |38| 10-K ledger funnel | 143,704 | 341,207 − 188,518 − 35,037 − 1,112 = 116,540 + 27,164 legacy = 143,704 | = |39| universe | 2,573 → 2,555 → 2,564; 7 unmatched | 2,573 lines − 18 dual-class CIKs = 2,555; 500 + 2,555 − 491 = 2,564; 7 unmatched | = |40| first-seen range 2011–2025 | "6,200–8,200" | 6,240–8,216 | = |41| `confidence = 0.600` | (Agent 8 probe) 3,278 | 3,278 | = |42| gate (DuckDB RE2 vs Python `re`) | — | 703,737 = 703,737; 0 productive sections fail | = |43| `_focus` replica vs real function | — | 0 mismatches on 1,123,098 sections | = |44| 28 baseline CSV exports | committed 678d233 | 26 byte-identical; `projects_by_index.csv`, `top_issuers.csv` identical as sets, rows reordered among ties (no deterministic ORDER BY in the SQL) | = (order) |4546## 3. What changed (the numbers the manuscript must adopt; details in `implemented_changes.md`, "N-batches")4748- **"Sections submitted" → true calls:** 703,725 of 1,123,098 (419,361 rows never called); productive share 23.9 % of calls (15.0 % of rows); 0.44 mentions per call; ≈ 0.0013 USD per call.49- **Silent failures:** 14 JSON-decode failures outside the outage (paper: "one"), June 3; and **one post-resume batch of 5,198 sections (2,686 calls) with 23 productive vs 536 expected and no logged failure** — suspect block, settable only by re-submission (AR-LLM-4 proposed in `unresolved_issues.md`).50- **Throughput:** 13.68 h wall-clock, 5.25 h gap, 31.9 rows/s = 20.1 calls/s effective (paper: "about twelve hours, 27–36/s").51- **Project count:** 121,099 under the current key; **range 96,749 (type dropped) – 198,488 (strict full-name + location), 21,027 under cik|type, 308,065 at mention level**; 54.8 % of projects keyed on a name token, `acquisition` alone absorbing 40,407 mentions; 33,820 projects have arg_max ties (394 names / 600 descriptions differ between the authors' run and ours).52- **Grounding (new):** years 99.8 %, partners 99.4 %, amount number 93.7 % (80.5 % with currency within 40 chars), locations 78.3 %, technology tags 63.6 %; all structured fields 88.2 % (8-K 76.6 %); regex-backfill upper bounds 39 % (years) / 46 % (technologies).53- **Focus coverage (new):** median shown fraction 9.3 % of a 10-K MD&A, last window at 16 %; 62.7 % of mentions from sections shown < 25 %; mention yield plateaus at ≈ 2.5 per 10,000 shown characters; 0 messages cut at 14,000; only 169 (0.6 %) MD&As < 30 words (the "2,081" is the legacy substitution).54- **Two runs / two segmenters:** no June git revision; extractor files dated 2026-06-17; legacy MD&A text has half the money tokens of the parsed text; 86 `full_document` 10-Ks (35 invisible); cohort effect +8.8 → −23.0 projects per issuer when the run is held fixed; `fiscal_year` NULL for 68,827 mentions with no stored source to fill it.55- **Construct violations:** 7.3 % of mentions match the financing/buyback class (54.5 % of 10-Q unregistered_sales, 52.6 % of capex_program).56- **Amounts:** max 10¹² (1 row), 161 mentions ≥ 50 B (104 projects), 584 zeros with `amount_raw` NULL, 24 negatives; `amount_raw` is a formatted float with 584 deviations (not 1).57- **8-K proxy (new):** P(M&A mention | Item 2.01) = 0.91 vs 0.03 (Item 5.02); P(cost_reduction | Item 2.05) = 0.71.58- **Fixed cohort:** new projects per issuer 4.7 → 2.8 (2011 → 2025) with mentions per issuer flat (9.6 → 8.8).59- **Universe:** holdings CSV and `company_tickers.json` not archived (2,587 / 14 / 21 not reconstructible); `annual_panel` has 2,499 CIKs (paper 2,501); 2 filings missing on disk (XEL 10-K 2014, LNT 10-Q 2012), 14 parse errors.6061## 4. Rules baseline and annotation frame (N8, N11)6263N8 (200 s, stratified seeded sample 30,000 + 30,000 sections; a first run had drawn a simple random sample because DuckDB applies `USING SAMPLE` before `WHERE` — corrected with subqueries and re-run): P(rules ≥ 1 | LLM ≥ 1) = 85.7 % (10-K 97.4, 10-Q 94.4, 8-K 69.2), weighted P(LLM ≥ 1 | rules ≥ 1) = 32.5 %, agreement 71.2 %, type overlap on co-detected sections 84.6 % (M&A 96.8 %); rules on `_focus(text)` recover 26.2 % of the rules-on-full-text mentions in over-budget sections (10-K MD&A 20.8 %). N11 (9 s): 1,677 productive sections in 192 strata (+ core 510) and 385 zero-mention sections, weights and dev/holdout split, ± 4.4 pp / ± 2.4 pp design precision; frames with full text on the node, protocol in `docs/annotation_protocol.md`; no label produced.6465## 5. Export regeneration (N13) and reproducibility verdict6667`code/export_stats.py` (L1 revision `64ddafa`, sha256 a7bf4939…) re-run on M3U96b in 44 s into a scratch directory: 26 of the 28 baseline CSVs byte-identical, 2 identical up to the order of tied rows, `stats.json` 17 → 41 keys (`example_project` removed as intended), 4 new files (`calls_by_form.csv`, `cohort_by_run.csv`, `ledger_collisions.csv`, `projects_by_type_ci.csv`). The regenerated `stats.json` and the 4 new files are copied into `data/`; the 2 reordered files are left as committed (content identical). Cross-checks against the N audits: `gate_pass_total` 703,737 = N1; `calls_by_form` = N1; `backfilled_rows` 154,964 / `_under_30_words` 1,070 = N1; `outage` 01:25:35 → 06:40:45, 5.25 h = N1; `conf_exact_default_count` 3,278 = N1; `ledger_10k_mdna_median_words` 9,905 = N5; `annual_panel_firms` 2,499 = N10; `amount_ge_50b` 104 projects = N2. **Four values of the L1 script disagree with the audits and must be corrected by its owner before Stage E uses them:** `amount_raw_deviations` = 0 (the `<>` comparison is NULL-blind; `IS DISTINCT FROM` gives 584), `october_wall_clock_hours` = 22.05 and `october_effective_sections_per_second` = 16.0 (measured from the backfill timestamp 13:12 instead of the first October flush 21:35; correct 13.68 h and 31.9 rows/s), `spid_runs.sp500_run.sections_logged` = 4,271 (first batch, not 153,894). Deterministic ordering (`ORDER BY … , cik` / `, ticker`) is recommended for `projects_by_index` and `top_issuers`.6869**Verdict for the data layer: R1** — the exports are reproduced from the documented analysis data (the two DuckDB files on the master node; raw filings 254.5 GB not needed), with the two order-only differences and the four flagged `stats.json` values listed above. The audit numbers of N1–N11 are new measurements, each produced by a committed script with a manifest; none changes a conclusion of the paper's thesis ("research database under validation, gate (d) not met"); several weaken positive statements as Rule 2 requires (R4 not triggered).70