1# REPRODUCIBILITY23| Artefact | Command | Inputs | Deterministic? |4|---|---|---|---|5| corpus manifest + audit | `make audit` | PDFs, index.csv, Crossref (cached in crossref_cache.json) | yes (given cache) |6| parsed documents | `make parse` | PDFs | yes (parser version in each file) |7| parser QA | `.venv/bin/python -I scripts/02_parser_qa.py` | parsed + pdftotext | yes (seed 45) |8| splits | `.venv/bin/python -I scripts/03_make_splits.py` | manifest | yes (seed 45) |9| annotations | `make annotate` | parsed, API | **no** (LLM; responses cached by prompt hash under data/annotations/_llm_cache) |10| profiles + figures | `make profiles` | parsed, annotations, split | yes (bootstrap seed 45) |11| index | `make index` | parsed, annotations | yes (same embedding model) |12| fixtures | `.venv/bin/python -I scripts/04_build_fixtures.py` | parsed, annotations, split | yes (seed 45) |13| benchmark | `jof250 benchmark --config config/evaluation.yaml` | fixtures, API | **no** (LLM; all raw runs stored) |14| analysis + paper tables/figures | `.venv/bin/python -I scripts/05_analyze_benchmark.py` | runs.jsonl, judgments.jsonl | yes (seed 45) |15| paper | `cd paper && make` | sections, tables, figures, references.bib | yes |1617Every JSON artefact carries `timestamp`, `code_version` (git short SHA, `-dirty` if uncommitted) and `host`. API calls: `data/annotations/api_ledger.jsonl` (model, prompt id/version, tokens, cost, request id). Environment: `pyproject.toml`; Python 3.12.13; model ids in `config/models.yaml`.18