Implementation audit — UQO WP 28 (checked-out revision 2324b58, 2026-10-28)
Repository: gitsrv:srv/git/uqo-research-db.git (private UQO Git server; the public alias cited in the brief, git.spboucher.ai/uqo-research-db.git, points to the same server). The June 2026 methodological study examined df7905d; the paper's numbers come from the build at 2324b58.
Inspected (paths as checked out)
src/acquisition/{universe,bulk_downloader,cik_mapper,sec_client}.py,scripts/{build_universe,download_filings,build_database,parse_filings,build_embeddings,build_factors,export_shard,merge_shards}.py,scripts/cluster/{orchestrate.py,rebalance.py,run_shard.sh,spid_run.sh}project-db/spid/{taxonomy,pipeline,resolve,graph,embed,store,config}.py,project-db/spid/extract/{base,rules,llm}.py,project-db/run_full_extract.py- Artifacts:
data/metadata/universe.json,shards.json,download_log*.jsonl, the two DuckDB files,data/logs/spid_extract.log(outage window 2026-10-28 01:10 → 06:22) - No
AGENTS.md;CLAUDE.mdfiles exist only at the user level. No unit tests exist for the SPID package.
Evidence-backed discrepancies vs. the brief
| # | Finding | Evidence | Priority |
|---|---|---|---|
| 1 | No human-labelled gold standard; precision/recall unknown | no human_labels table, no annotation files |
High |
| 2 | Resolution key merges distinct phases / splits renamed projects; no link scores | resolve.py (MD5 of cik |
type |
| 3 | extracted_sections cannot distinguish "no project" from "API failure" |
149,352 zero-mention rows written during the credit outage, removed by timestamp | High |
| 4 | Amounts untyped; project amount = max over mentions | resolve.py; amount_raw kept |
High |
| 5 | No project_events; timeline = mention rows; cited years unordered |
project_timeline schema |
Medium |
| 6 | No prompt / taxonomy / schema hash per run | extracted_sections.backend = "llm" only |
Medium |
| 7 | Evidence = model paraphrase; no character span of the supporting passage | project_mentions.description |
Medium |
| 8 | Focus windows truncate long sections (budget ≈ 9–12.6k chars) — recall on 10-K MD&A unknown | llm.py::_focus |
Medium |
| 9 | Universe: no file SHA-256, no match confidence, no membership dates, no security-vs-issuer table | universe.json |
Medium |
| 10 | Form scope excludes 8-K exhibits (press releases) and 20-F/40-F (65 issuers) | download_filings.py |
Medium |
| 11 | Sections lack offsets into raw HTML (chunks have offsets within sections) | filing_sections schema |
Low |
| 12 | pq.write_to_dataset(overwrite_or_ignore) leaves stale partitions → duplicates in filing_scores until cleaned (done 2026-10-28) |
aggregation.py |
Low |
| 13 | Partner nodes of the knowledge graph have no edges | graph.py |
Low |
Prioritised plan
- Annotation study (protocol in paper §8): sample, two annotators, adjudication, locked holdout → metrics.
- Schema migrations (additive):
mention_label,evidence_span,call_statusin the ledger,extraction_runswith hashes,project_mention_links,project_events,project_amounts,human_labels. - Probabilistic linker with reviewable clusters; re-derive
projectsfrom links. - Second extraction pass for
amount_type; 8-K exhibit 99 ingestion; recall test on long MD&As with larger focus budgets. - Universe hardening (hashes, confidence, security table); evaluate a licensed historical constituent list.