Repository Public Research preview
1# Reproducibility23Version 0.1 · 2026-11-09. All commands run from the repository root. `PY=.venv/bin/python`.45## 1. Environment67**Reference machine** (recorded in every run header):89- Apple M5 Max, 48 GB, macOS 27.0, arm64, 18 CPUs;10- Python 3.14.4;11- `.venv` on Homebrew Python with system site-packages.1213**Library versions** (logged by `htr.utils.provenance.run_header`):1415| Library | Version |16|---|---|17| torch | 2.12.0 (MPS) |18| numpy | 2.5.4 |19| anthropic | 2.3.0 |20| transformers | 5.8.1 |21| sentencepiece | 0.2.2 |22| lxml | 6.1.3 |23| pyarrow | 24.0.0 |24| sacrebleu | 2.6.0 |25| datasketch | 2.0.0 |26| rapidfuzz | 3.14.6 |2728**Other packages present on the reference machine:**2930- sentence-transformers 5.5.0, with `all-MiniLM-L6-v2` cached locally. It is loaded with `HF_HUB_OFFLINE=1`31 and never downloaded.32- PyMuPDF 1.27.33- pandoc and TeX Live (xelatex, latexmk, biber) for the paper.3435**Clean environment:**3637```bash38python3 -m venv .venv39.venv/bin/pip install -e ".[api,hf,dev]" # pyproject.toml; Python >= 3.1240.venv/bin/python -m pytest tests -q -m "not network"41```4243**Run headers.** Every summary or manifest JSON starts with a run header containing `created_at` (UTC),44`git{commit,branch,dirty}`, `libraries` and `hardware`. The pilot artefacts record a **dirty** tree45(`git.dirty = true`). Confirmatory runs must be made from a clean, committed tree. A run with `dirty = true`46is development evidence only.4748## 2. Pipeline commands4950| Step | Command | Output |51|---|---|---|52| Vertical slice (zero cost) | `scripts/vertical_slice.sh [data/parsed/smoke] [120]` | splits, `data/pairs/vs_<name>/`, tokenizer, `artifacts/runs/vs_<name>-m1-nano/`, predictions, metrics |53| PMC discovery + fetch | `$PY scripts/discover_pmc.py --candidates 6000 --max-docs 1200 --seed 2 --out data/raw/pmc` | `data/raw/pmc/*.xml` + `.meta.json`; manifest `data/manifests/discover_pmc_seed2.json` |54| arXiv (in progress) | `$PY scripts/discover_arxiv.py --sets q-fin econ --from 2008-01-01 --until 2017-12-31 --max-docs 12 --seed 1` | `data/raw/arxiv/` |55| Books | `$PY scripts/discover_books.py --source gutenberg --max-docs 12 --seed 1 --out data/raw/books` (also `--source bookshelf`, `--source openstax`) | `data/raw/books/`, `data/manifests/discover_books_*` |56| Crossref (in progress) | script not yet present | — |57| Build corpus | `$PY scripts/build_corpus.py --raw data/raw/pmc --out data/parsed/pilot_pmc --tag pilot_pmc` | `documents.jsonl`, `passages.jsonl`, `excluded.jsonl`, `clusters.json`, `build_summary.json` |58| Splits | `$PY scripts/make_splits.py --corpus data/parsed/pilot_pmc --heldout-disciplines physics --heldout-before-year 1990 [--fixed old/splits.jsonl]` | `data/splits/<name>/{splits.jsonl,splits_summary.json}` |59| Rewrite: estimate | `$PY scripts/rewrite.py --corpus data/parsed/pilot_pmc --job pilot_opus55 --n-passages 1000 --seed 1 --splits data/splits/pilot_pmc/splits.jsonl --estimate-only` | prints `estimate:`; writes the job `manifest.json` |60| Rewrite: mock | the same command without `--estimate-only` | deterministic mock rewrites, 0 USD |61| Rewrite: live | `... --live [--max-requests 200]` | real API calls; ledger; stops at the ceiling |62| Rewrite: batch | `... --live --batch [--max-requests 200]`, then later `... --live --collect` | `batches/<id>.json`, then `rewrites.jsonl` |63| Held-out model / prompt | `... --live --batch --model claude-sonnet-5-5` · `... --variants H1,H2` | separate job directories |64| Validate | `$PY scripts/validate_pairs.py --corpus data/parsed/pilot_pmc --job pilot_opus55 [--no-embeddings]` | `validated/{pairs,quarantine}.jsonl`, `validation_summary.json` |65| Tokenizer | `$PY scripts/train_tokenizer.py --pairs <pairs.jsonl> --splits <splits.jsonl> --out artifacts/tokenizers/<name> --vocab 32000` | `tokenizer.model`, `tokenizer_config.json`, summary (train split only) |66| Train | `$PY scripts/train.py --pairs … --splits … --tokenizer artifacts/tokenizers/<name> --run <run> --preset {m1,m2,m3,m3_*} --size {nano,tiny,small,base,large} --seed 1 [--overfit-test] [--resume]` | `artifacts/runs/<run>/{manifest.json,train_log.jsonl,train_summary.json,best/,last/}` |67| Predict | `$PY scripts/predict.py --pairs … --splits … --split test --checkpoint artifacts/runs/<run>/best --baselines --out artifacts/predictions/<name> [--beam 4]` | `outputs_<system>.jsonl` |68| Evaluate | `$PY scripts/evaluate.py --pairs … --outputs-dir artifacts/predictions/<name> --name <name>` | `artifacts/metrics/<name>/{report.json,summary.md,items_*.jsonl}` |6970**Notes on the commands.**7172- **Rewrite selection.** `rewrite.py` selects passages stratified by (discipline, section role) with73 `--seed`. Only splits listed in `--allowed-splits` are eligible.74- **Spend.** No money is spent without `--live`.75- **Corpus default path.** The `--corpus` default in several scripts is `data/parsed/pilot`. The existing76 pilot directory is `data/parsed/pilot_pmc`, so pass it explicitly.7778## 3. Seeds and determinism7980| What | Seed |81|---|---|82| PMC candidate sampling | `--seed` (manifests seed 1 and 2) |83| Split assignment | sha256(`seed:group`) with seed 20261109; deterministic, independent of input order |84| Rewrite selection | `--seed` (1) |85| Mock provider | `seed` |86| Training | `torch.manual_seed(seed)`, bucket-sampler shuffle `seed + epoch` |87| Bootstrap | `seed = 0` |8889**What else is deterministic.**9091- Request ids are deterministic, so re-running a job skips completed requests.92- Inference is deterministic: greedy or beam, no sampling.93- MPS kernels are not guaranteed bit-identical across OS or torch versions. Report three training seeds instead94 of relying on bitwise reproducibility.9596## 4. Data hashes and provenance9798**Per document:**99100- `raw_sha256`;101- the provider MD5 and its verification (`provider_md5_ok`; 0 failures in 1 300 fetches);102- the licence record with its evidence;103- the date with its evidence;104- `acquired_at`.105106**Other identifiers and hashes:**107108- Passages: `text_sha256`, and ids derived from (doc, paragraph index, text hash).109- Rewrites: provider, model, `prompt_version` (`rw-1.0`), prompt fingerprint (sha256 of system + preserve +110 variants), request id, usage and cost.111- Pairs: `validator_version` (`val-1.0`).112- Gutenberg catalog: sha256 recorded in its manifest.113114## 5. What is committed and what is not115116| Committed | Not committed (`.gitignore`) |117|---|---|118| code, configs, tests, docs, `data/manifests/*.json`, `artifacts/{metrics,figures,tables,reports}`, `paper/` sources and `paper/main.pdf` | `data/{raw,parsed,segments,pairs,splits}/*`, `artifacts/checkpoints/*`, `artifacts/runs/`, `.env`, keys |119120**Consequence.** Third parties regenerate the data from the manifests and the commands above. Release-eligible121parsed text is distributed only through the benchmark bundle (`LICENSE_MATRIX.md`). The API key is read from122the environment (`ANTHROPIC_API_KEY`) and is never written to disk by the code.123124## 6. From a clean environment to Gate F1251261. Clone `gitsrv:srv/git/wpai0002_uqo.git`.1272. Create the venv as in §1 and run the unit tests.1283. Run `scripts/vertical_slice.sh` (no network, no cost) to check the pipeline end to end.1294. Load a released checkpoint:130 ```python131 HTRReconstructor.from_pretrained(dir).reconstruct(text, metadata)132 ```133 Or run `scripts/predict.py --checkpoint <dir>`.1345. Gate F re-runs step 4 on a machine that never saw the training data, and compares outputs bit-for-bit with135 the release's reference outputs on CPU.136137## Known test-environment issue (2026-11-09)138139On the laptop, running the full unit suite while a PyTorch MPS training process is active can crash the test interpreter with a native stack trace in `embedding` (observed with torch 2.12 / Python 3.14 when `tests/unit/test_models.py` runs after another module; `test_models.py` alone passes). The suite passes (47 tests) when no MPS training is running. Run `pytest` on an idle machine or on CPU-only nodes; this is an environment interaction, not a test failure of the code under test.140