spb/firc0001_uqo

1# Implementation audit — UQO Working Paper No. 0001 (checked-out revision 2324b58, 2026-10-28; annotated 2026-11-05 after peer review)23Repository: `gitsrv:srv/git/uqo-research-db.git` (private UQO Git server; the public alias cited in the brief, `git.spboucher.ai/uqo-research-db.git`, points to the same server). The June 2026 methodological study cited `df7905d` as its revision, but that commit is dated 2026-10-27 and the repository's six commits are all of 2026-10-27/28: no June revision exists (file-system evidence only, `results/runs/git_evidence.md`). The paper's numbers come from the build at `2324b58`.45## Inspected (paths as checked out)6- `src/acquisition/{universe,bulk_downloader,cik_mapper,sec_client}.py`, `scripts/{build_universe,download_filings,build_database,parse_filings,build_embeddings,build_factors,export_shard,merge_shards}.py`, `scripts/cluster/{orchestrate.py,rebalance.py,run_shard.sh,spid_run.sh}`7- `project-db/spid/{taxonomy,pipeline,resolve,graph,embed,store,config}.py`, `project-db/spid/extract/{base,rules,llm}.py`, `project-db/run_full_extract.py`8- Artifacts: `data/metadata/universe.json`, `shards.json`, `download_log*.jsonl`, the two DuckDB files, `data/logs/spid_extract.log` (outage window 2026-10-28 01:10 → 06:22)9- No `AGENTS.md`; `CLAUDE.md` files exist only at the user level. No unit tests exist for the SPID package.1011## Evidence-backed discrepancies vs. the brief12| # | Finding | Evidence | Priority |13|---|---|---|---|14| 1 | No human-labelled gold standard; precision/recall unknown | no `human_labels` table, no annotation files | High |15| 2 | Resolution key merges distinct phases / splits renamed projects; no link scores | `resolve.py` (MD5 of cik|type|loc/token); Cheniere example | High |16| 3 | `extracted_sections` cannot distinguish "no project" from "API failure" (nor from a gate rejection: 419,361 rows never sent) | 149,352 zero-mention rows written during the credit outage, removed by timestamp (reconciled with the log, N1); 14 silent JSON-decode failures outside the outage; first post-outage batch (5,198 sections, 23 productive vs 536 expected) suspect — re-submission AR-LLM-4 not granted | High |17| 4 | Amounts untyped; project amount = max over mentions; `amount_raw` is a formatted copy of the coerced float, not the source string | `resolve.py`; `llm.py:155`; 584 zero amounts with NULL `amount_raw`, 24 negative, 161 mentions ≥ 50 B USD (N1/N2) | High |18| 5 | No `project_events`; timeline = mention rows; cited years unordered | `project_timeline` schema | Medium |19| 6 | No prompt / taxonomy / schema hash per run | `extracted_sections.backend` = "llm" only | Medium |20| 7 | Evidence = model paraphrase; no character span of the supporting passage | `project_mentions.description` | Medium |21| 8 | Focus windows truncate long sections (budget ≈ 9–12.6k chars) — measured 2026-11-05: median 9.3 % of a 10-K MD&A shown, 62.7 % of mentions from sections shown < 25 %, rules-based recall bound 26.2 % | `llm.py::_focus`; `results/focus/`, `results/rules/` (N5, N8) | Medium |22| 9 | Universe: derived artefacts now hashed (N10) but source holdings/ticker files not archived; no match confidence, no membership dates, no security-vs-issuer table | `universe.json`, `data/metadata/SHA256SUMS` | Medium |23| 10 | Form scope excludes 8-K exhibits (press releases) and 20-F/40-F (65 issuers) | `download_filings.py` | Medium |24| 11 | Sections lack offsets into raw HTML (chunks have offsets within sections) | `filing_sections` schema | Low |25| 12 | `pq.write_to_dataset(overwrite_or_ignore)` leaves stale partitions → duplicates in `filing_scores` until cleaned (done 2026-10-28) | `aggregation.py` | Low |26| 13 | Partner nodes of the knowledge graph have no edges | `graph.py` | Low |2728## Prioritised plan291. Annotation study (protocol in paper §8): sample, two annotators, adjudication, locked holdout → metrics.302. Schema migrations (additive): `mention_label`, `evidence_span`, `call_status` in the ledger, `extraction_runs` with hashes, `project_mention_links`, `project_events`, `project_amounts`, `human_labels`.313. Probabilistic linker with reviewable clusters; re-derive `projects` from links.324. Second extraction pass for `amount_type`; 8-K exhibit 99 ingestion; recall test on long MD&As with larger focus budgets.335. Universe hardening (hashes, confidence, security table); evaluate a licensed historical constituent list.34