Implementation audit — UQO Working Paper No. 0001 (checked-out revision 2324b58, 2026-10-28; annotated 2026-11-05 after peer review)
Repository: gitsrv:srv/git/uqo-research-db.git (private UQO Git server; the public alias cited in the brief, git.spboucher.ai/uqo-research-db.git, points to the same server). The June 2026 methodological study cited df7905d as its revision, but that commit is dated 2026-10-27 and the repository's six commits are all of 2026-10-27/28: no June revision exists (file-system evidence only, results/runs/git_evidence.md). The paper's numbers come from the build at 2324b58.
Inspected (paths as checked out)
src/acquisition/{universe,bulk_downloader,cik_mapper,sec_client}.py,scripts/{build_universe,download_filings,build_database,parse_filings,build_embeddings,build_factors,export_shard,merge_shards}.py,scripts/cluster/{orchestrate.py,rebalance.py,run_shard.sh,spid_run.sh}project-db/spid/{taxonomy,pipeline,resolve,graph,embed,store,config}.py,project-db/spid/extract/{base,rules,llm}.py,project-db/run_full_extract.py- Artifacts:
data/metadata/universe.json,shards.json,download_log*.jsonl, the two DuckDB files,data/logs/spid_extract.log(outage window 2026-10-28 01:10 → 06:22) - No
AGENTS.md;CLAUDE.mdfiles exist only at the user level. No unit tests exist for the SPID package.
Evidence-backed discrepancies vs. the brief
| # | Finding | Evidence | Priority |
|---|---|---|---|
| 1 | No human-labelled gold standard; precision/recall unknown | no human_labels table, no annotation files |
High |
| 2 | Resolution key merges distinct phases / splits renamed projects; no link scores | resolve.py (MD5 of cik |
type |
| 3 | extracted_sections cannot distinguish "no project" from "API failure" (nor from a gate rejection: 419,361 rows never sent) |
149,352 zero-mention rows written during the credit outage, removed by timestamp (reconciled with the log, N1); 14 silent JSON-decode failures outside the outage; first post-outage batch (5,198 sections, 23 productive vs 536 expected) suspect — re-submission AR-LLM-4 not granted | High |
| 4 | Amounts untyped; project amount = max over mentions; amount_raw is a formatted copy of the coerced float, not the source string |
resolve.py; llm.py:155; 584 zero amounts with NULL amount_raw, 24 negative, 161 mentions ≥ 50 B USD (N1/N2) |
High |
| 5 | No project_events; timeline = mention rows; cited years unordered |
project_timeline schema |
Medium |
| 6 | No prompt / taxonomy / schema hash per run | extracted_sections.backend = "llm" only |
Medium |
| 7 | Evidence = model paraphrase; no character span of the supporting passage | project_mentions.description |
Medium |
| 8 | Focus windows truncate long sections (budget ≈ 9–12.6k chars) — measured 2026-11-05: median 9.3 % of a 10-K MD&A shown, 62.7 % of mentions from sections shown < 25 %, rules-based recall bound 26.2 % | llm.py::_focus; results/focus/, results/rules/ (N5, N8) |
Medium |
| 9 | Universe: derived artefacts now hashed (N10) but source holdings/ticker files not archived; no match confidence, no membership dates, no security-vs-issuer table | universe.json, data/metadata/SHA256SUMS |
Medium |
| 10 | Form scope excludes 8-K exhibits (press releases) and 20-F/40-F (65 issuers) | download_filings.py |
Medium |
| 11 | Sections lack offsets into raw HTML (chunks have offsets within sections) | filing_sections schema |
Low |
| 12 | pq.write_to_dataset(overwrite_or_ignore) leaves stale partitions → duplicates in filing_scores until cleaned (done 2026-10-28) |
aggregation.py |
Low |
| 13 | Partner nodes of the knowledge graph have no edges | graph.py |
Low |
Prioritised plan
- Annotation study (protocol in paper §8): sample, two annotators, adjudication, locked holdout → metrics.
- Schema migrations (additive):
mention_label,evidence_span,call_statusin the ledger,extraction_runswith hashes,project_mention_links,project_events,project_amounts,human_labels. - Probabilistic linker with reviewable clusters; re-derive
projectsfrom links. - Second extraction pass for
amount_type; 8-K exhibit 99 ingestion; recall test on long MD&As with larger focus budgets. - Universe hardening (hashes, confidence, security table); evaluate a licensed historical constituent list.