SPB Git

spb/wp45_uqo

Public
2.5 KB

# ARCHITECTURE

text
PDFs (read-only) ──inventory.py──▶ data/manifests/{corpus_manifest.parquet, corpus_v1_frozen.json, corpus_audit.md}
        │                                   (SHA-256, pages, validity, index.csv join, Crossref verification, decisions)
        └──parsing.py (PyMuPDF layout heuristics v1.3)──▶ data/derived/parsed/<doc_id>.json   [private]
                 title/authors/abstract, sections (roman numerals) + subsections, paragraphs with pages, numbered equations,
                 tables (incl. rotated), figure captions, footnotes, references, in-text citations, confidence per component
        └──segmentation.py: section roles (rule-based) + bounded annotation payload (rights.yaml cap)
        └──rhetoric.py: ontology v1.0 (JSON schema, strict local validation) ← llm.py (Claude API, ledger, budget cap, cache)
                 → data/annotations/documents/<doc_id>.json, annotations_table.parquet
        └──features.py: train-split features → profiles/*.yaml (+ figures/)     [heldout_v1 split: data/splits/]
        └──embeddings.py: units (summaries, identification, intro moves, claim–evidence, section openings, table notes, captions ≤400 chars)
                 → data/derived/index/{units.parquet, embeddings.npy (MiniLM-L6, local), index_meta.json}
        └──retrieval.py: hybrid BM25 + cosine, filters (split/year/genre/subfield/role/unit), ≤2 units per doc, provenance
generation.py: manuscript loader (auditing.load_manuscript) → genre/subfield → profile → retrieval → draft (conditions A–E)
               → deterministic audit (auditing.py) → specialist referee pass (E) → new file + .report.md; engine_runs.jsonl
auditing.py:  tables/equations/labels, citation keys + Crossref DOI resolution, numeric preservation, unsupported claims,
               8-gram corpus overlap, evidence lookup — no LLM
evaluation.py: preregistered protocol heldout_v1 (fixtures from held-out docs + WP29 OOD; conditions + ablations; blind judge)
integrations/claude_code.py: .claude/{commands,agents,skills} generator; user-scope install on request
cli.py (Typer `jof250`), reporting.py (status), provenance.py (hashes, stamps, ledgers), config.py

Design choices: local-first (PyMuPDF, DuckDB/Parquet-compatible artefacts, local embeddings), no server dependencies (GROBID/Docker unavailable on the host), every artefact carries timestamp, code_version, host; copyrighted full texts stay in gitignored directories; API calls are prefix-cached (system prompt) and content-cached by hash.

Files · main
Menu

Theme