REPRODUCIBILITY
| Artefact | Command | Inputs | Deterministic? |
|---|---|---|---|
| corpus manifest + audit | make audit |
PDFs, index.csv, Crossref (cached in crossref_cache.json) | yes (given cache) |
| parsed documents | make parse |
PDFs | yes (parser version in each file) |
| parser QA | .venv/bin/python -I scripts/02_parser_qa.py |
parsed + pdftotext | yes (seed 45) |
| splits | .venv/bin/python -I scripts/03_make_splits.py |
manifest | yes (seed 45) |
| annotations | make annotate |
parsed, API | no (LLM; responses cached by prompt hash under data/annotations/_llm_cache) |
| profiles + figures | make profiles |
parsed, annotations, split | yes (bootstrap seed 45) |
| index | make index |
parsed, annotations | yes (same embedding model) |
| fixtures | .venv/bin/python -I scripts/04_build_fixtures.py |
parsed, annotations, split | yes (seed 45) |
| benchmark | jof250 benchmark --config config/evaluation.yaml |
fixtures, API | no (LLM; all raw runs stored) |
| analysis + paper tables/figures | .venv/bin/python -I scripts/05_analyze_benchmark.py |
runs.jsonl, judgments.jsonl | yes (seed 45) |
| paper | cd paper && make |
sections, tables, figures, references.bib | yes |
Every JSON artefact carries timestamp, code_version (git short SHA, -dirty if uncommitted) and host. API calls: data/annotations/api_ledger.jsonl (model, prompt id/version, tokens, cost, request id). Environment: pyproject.toml; Python 3.12.13; model ids in config/models.yaml.