Add arXiv/LaTeX, Crossref/PDF and books adapters (agents), governance docs, KV-cache decoding, release/CLI/demo, B2 baseline, paper skeleton, pilot Gate B artefacts
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
17 changed files +19,994 −1
modified
CLAUDE.md
+2 −1
@@ -25,7 +25,8 @@ Operational rules that follow from the mandate: | |||||
| 25 | 4. **No leakage.** Splits are by document, near-duplicate cluster and author group (`htr.data.splits`); held-out | 25 | 4. **No leakage.** Splits are by document, near-duplicate cluster and author group (`htr.data.splits`); held-out | ||
| 26 | discipline, period and rewriting prompt; originals of held-out items are never used at inference. | 26 | discipline, period and rewriting prompt; originals of held-out items are never used at inference. | ||
| 27 | 5. **Spend is opt-in.** The Anthropic adapter runs in `mock`/`dry-run` by default; real calls need | 27 | 5. **Spend is opt-in.** The Anthropic adapter runs in `mock`/`dry-run` by default; real calls need | ||
| 28 | − `--live` and a budget ceiling; the pilot ceiling is in `configs/rewriting/*.yaml` and `docs/COMPUTE_BUDGET.md`. | 28 | + `--live` and a budget ceiling; the pilot ceiling is in `configs/rewriting/pilot.json` and | ||
| 29 | + `docs/reproducibility/COMPUTE_BUDGET.md`. Spend so far is in `docs/STATUS.md` (from the job ledgers). | ||||
| 29 | Ask before exceeding it, before mass acquisition beyond the planned stages, before any public upload. | 30 | Ask before exceeding it, before mass acquisition beyond the planned stages, before any public upload. | ||
| 30 | 6. **Gates in order** (A repository + governance + 100-paper smoke → B validated pairs → C splits/baselines → | 31 | 6. **Gates in order** (A repository + governance + 100-paper smoke → B validated pairs → C splits/baselines → | ||
| 31 | D M1/M2/M3 pilot → E scale → F release + paper). Each gate writes `docs/gates/GATE_<X>.md` with GO/NO-GO. | 32 | D M1/M2/M3 pilot → E scale → F release + paper). Each gate writes `docs/gates/GATE_<X>.md` with GO/NO-GO. | ||
added
README.md
+101 −0
@@ -0,0 +1,101 @@ | |||||
| 1 | +# WP-AI-0002 — Human Text Reconstruction Transformer (HTR) | ||||
| 2 | + | ||||
| 3 | +**Question.** Can a Transformer reconstruct characteristics of pre-2018 human-authored scientific prose from | ||||
| 4 | +LLM-rewritten versions of the same passages? | ||||
| 5 | + | ||||
| 6 | +**Deliverables.** The project, part of UQO Research (series AI, id AI-0002), has two deliverables: | ||||
| 7 | + | ||||
| 8 | +- a peer-review-ready paper, *Human Text Reconstruction: Learning the Inverse of LLM-Induced Transformations in | ||||
| 9 | + Scientific Writing*; | ||||
| 10 | +- a released model family (HTR-Tiny/Small/Base, plus Large if feasible) that can actually be used. | ||||
| 11 | + | ||||
| 12 | +**Prior work.** Paraphrase inversion already exists (Rivera Soto et al., 2024, 2026). This project studies | ||||
| 13 | +something different: | ||||
| 14 | + | ||||
| 15 | +- **Reconstruction fidelity** against identity and paraphrase baselines. | ||||
| 16 | +- **Rights-verified, pre-2018 scientific originals** with controlled Claude rewrite variants. | ||||
| 17 | +- **Factual integrity.** | ||||
| 18 | +- **Non-identifiability.** | ||||
| 19 | +- **Memorisation.** | ||||
| 20 | + | ||||
| 21 | +See `docs/RESEARCH_GAP.md`. | ||||
| 22 | + | ||||
| 23 | +## Status (2026-11-09) | ||||
| 24 | + | ||||
| 25 | +- **Gate A: GO.** 100-paper PMC smoke test with 2 013 eligible passages; licence governance in place | ||||
| 26 | + (`docs/gates/GATE_A.md`). | ||||
| 27 | +- **Pilot corpus.** 1 180 PMC documents with 23 658 eligible passages (`data/parsed/pilot_pmc`). Book trials: | ||||
| 28 | + 11 Gutenberg books and 17 NCBI Bookshelf chapters. arXiv and Crossref economics/finance adapters are in | ||||
| 29 | + progress. | ||||
| 30 | +- **Vertical slice.** Runs end to end with mock rewrites at 0 USD spend. M1-nano passes the one-batch overfit | ||||
| 31 | + test. | ||||
| 32 | +- **Not started:** real Claude rewrites (Gate B, ceiling 30 USD), trained models at release scale, and the paper | ||||
| 33 | + draft. | ||||
| 34 | +- **Full status:** `docs/IMPLEMENTATION_CHECKLIST.md`, `docs/architecture/RESEARCH_PLAN.md`. | ||||
| 35 | + | ||||
| 36 | +## Quick start | ||||
| 37 | + | ||||
| 38 | +```bash | ||||
| 39 | +python3 -m venv .venv && .venv/bin/pip install -e ".[api,hf,dev]" | ||||
| 40 | +.venv/bin/python -m pytest tests -q -m "not network" # unit tests, no network | ||||
| 41 | +scripts/vertical_slice.sh # corpus → mock rewrites → pairs → M1-nano → metrics (0 USD) | ||||
| 42 | +.venv/bin/python scripts/discover_pmc.py --candidates 400 --max-docs 100 --seed 1 --out data/raw/pmc_smoke | ||||
| 43 | +``` | ||||
| 44 | + | ||||
| 45 | +Using a trained checkpoint: | ||||
| 46 | + | ||||
| 47 | +```python | ||||
| 48 | +from htr.inference.pipeline import HTRReconstructor | ||||
| 49 | +r = HTRReconstructor.from_pretrained("artifacts/runs/<run>/best").reconstruct(text, {"discipline": "economics"}) | ||||
| 50 | +print(r.reconstruction, r.warnings) | ||||
| 51 | +``` | ||||
| 52 | + | ||||
| 53 | +**Spending money.** Real API calls require `--live` and respect the budget ceilings | ||||
| 54 | +(`docs/reproducibility/COMPUTE_BUDGET.md`). All commands are listed in | ||||
| 55 | +`docs/reproducibility/REPRODUCIBILITY.md`. | ||||
| 56 | + | ||||
| 57 | +## Layout | ||||
| 58 | + | ||||
| 59 | +``` | ||||
| 60 | +configs/ rewriting (pilot.json, pricing.json); data/training/evaluation/release (to fill) | ||||
| 61 | +src/htr/ ingest · licenses · parse · segment · dedup · data · rewrite · validate · models · training · | ||||
| 62 | + inference · evaluation · release · utils | ||||
| 63 | +scripts/ discover_{pmc,arxiv,books} · build_corpus · make_splits · rewrite · validate_pairs · | ||||
| 64 | + train_tokenizer · train · predict · evaluate · vertical_slice.sh | ||||
| 65 | +tests/ unit · integration (network-marked) · regression | ||||
| 66 | +data/ manifests (committed) · raw · parsed · pairs · splits (not committed) | ||||
| 67 | +artifacts/ runs/checkpoints (not committed) · metrics · figures · tables · reports | ||||
| 68 | +docs/ MANDATE · RESEARCH_GAP · PREREGISTRATION · DECISION_LOG · RISK_REGISTER · IMPLEMENTATION_CHECKLIST · | ||||
| 69 | + architecture/ · data_governance/ · experiments/ · reproducibility/ · gates/ · literature/ | ||||
| 70 | +paper/ model_cards/ demo/ | ||||
| 71 | +``` | ||||
| 72 | + | ||||
| 73 | +## Honest-claims notice | ||||
| 74 | + | ||||
| 75 | +HTR outputs are **synthetic reconstructions**. They are a model's estimate of pre-2018-style scientific prose | ||||
| 76 | +given an LLM rewrite. They are not the original text, they do not restore authorship, and they are not evidence | ||||
| 77 | +that a text was written by a human. HTR is not an AI-text detector. Every result is reported against identity | ||||
| 78 | +and paraphrase baselines with confidence intervals, and null or negative results are reported. | ||||
| 79 | + | ||||
| 80 | +## Licence | ||||
| 81 | + | ||||
| 82 | +- **Code:** Apache-2.0 (`pyproject.toml`). | ||||
| 83 | +- **Data and derived text:** per-document licences. Only CC0, public-domain and CC BY items are redistributable | ||||
| 84 | + (`docs/data_governance/LICENSE_MATRIX.md`). | ||||
| 85 | +- **Model weights:** licence to be decided at release. It depends on the training licence set and on | ||||
| 86 | + Anthropic's terms for model outputs (open question, `LICENSE_MATRIX.md` §4). | ||||
| 87 | +- **Attribution:** NLM/PMC and all source authors are acknowledged. | ||||
| 88 | + | ||||
| 89 | +## Citation | ||||
| 90 | + | ||||
| 91 | +```bibtex | ||||
| 92 | +@techreport{boucher2026htr, | ||||
| 93 | + author = {Boucher, Simon-Pierre}, | ||||
| 94 | + title = {Human Text Reconstruction: Learning the Inverse of {LLM}-Induced Transformations in Scientific Writing}, | ||||
| 95 | + institution = {Universit{\'e} du Qu{\'e}bec en Outaouais}, | ||||
| 96 | + type = {Working Paper}, | ||||
| 97 | + number = {AI-0002}, | ||||
| 98 | + year = {2026}, | ||||
| 99 | + note = {In preparation} | ||||
| 100 | +} | ||||
| 101 | +``` | ||||
added
artifacts/metrics/vs_smoke/items_B0_identity.jsonl
+26 −0
@@ -0,0 +1,26 @@ | |||||
| 1 | +{"pair_id": "pair_f83b9af962e6f241", "doc_id": "doc_fec5ce1b5f4440a9", "variant": "A", "discipline": "biomedicine", "era": "2010-2017", "role": "related_work", "exact_match": 0.0, "edit_sim": 0.5701943844492441, "chrf": 88.86673465164877, "bleu": 77.37201842777444, "rougeL": 0.7375886524822696, "len_ratio": 1.0142857142857142, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -1.0} | ||||
| 2 | +{"pair_id": "pair_e91a1b6883ac6e73", "doc_id": "doc_fec5ce1b5f4440a9", "variant": "B", "discipline": "biomedicine", "era": "2010-2017", "role": "related_work", "exact_match": 0.0, "edit_sim": 0.9436008676789588, "chrf": 91.43155358009184, "bleu": 86.38202910782107, "rougeL": 0.9275362318840579, "len_ratio": 0.9714285714285714, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": 0.0} | ||||
| 3 | +{"pair_id": "pair_a9af7ad1209bd3d3", "doc_id": "doc_fec5ce1b5f4440a9", "variant": "C", "discipline": "biomedicine", "era": "2010-2017", "role": "related_work", "exact_match": 0.0, "edit_sim": 0.9370932754880694, "chrf": 89.51649409867403, "bleu": 82.26760359337925, "rougeL": 0.920863309352518, "len_ratio": 0.9857142857142858, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -1.0} | ||||
| 4 | +{"pair_id": "pair_183ad7a4dd1ebebc", "doc_id": "doc_fec5ce1b5f4440a9", "variant": "D", "discipline": "biomedicine", "era": "2010-2017", "role": "related_work", "exact_match": 0.0, "edit_sim": 0.9392624728850325, "chrf": 91.14660547185507, "bleu": 86.38158557825156, "rougeL": 0.920863309352518, "len_ratio": 0.9857142857142858, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": 0.0} | ||||
| 5 | +{"pair_id": "pair_bf9bdf18c8d57734", "doc_id": "doc_fec5ce1b5f4440a9", "variant": "E", "discipline": "biomedicine", "era": "2010-2017", "role": "related_work", "exact_match": 0.0, "edit_sim": 0.8112798264642083, "chrf": 91.44536041111537, "bleu": 83.66869788571748, "rougeL": 0.8695652173913043, "len_ratio": 0.9714285714285714, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -1.0} | ||||
| 6 | +{"pair_id": "pair_13b7831e06faab4c", "doc_id": "doc_fec5ce1b5f4440a9", "variant": "E", "discipline": "biomedicine", "era": "2010-2017", "role": "related_work", "exact_match": 0.0, "edit_sim": 0.5184381778741866, "chrf": 94.40877580869888, "bleu": 89.20174339688546, "rougeL": 0.7101449275362319, "len_ratio": 0.9714285714285714, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": 0.0} | ||||
| 7 | +{"pair_id": "pair_008ecac459ef8ec7", "doc_id": "doc_2c8f18fd479cd367", "variant": "A", "discipline": "biomedicine", "era": "2000-2009", "role": "data", "exact_match": 0.0, "edit_sim": 0.8096234309623431, "chrf": 89.68307383622911, "bleu": 79.16556746602922, "rougeL": 0.8648648648648649, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -1.0} | ||||
| 8 | +{"pair_id": "pair_530abafa53fcf7d8", "doc_id": "doc_2c8f18fd479cd367", "variant": "B", "discipline": "biomedicine", "era": "2000-2009", "role": "data", "exact_match": 0.0, "edit_sim": 0.8977955911823647, "chrf": 89.70542901556904, "bleu": 78.23855754713819, "rougeL": 0.9139072847682119, "len_ratio": 1.0434782608695652, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": 0.0} | ||||
| 9 | +{"pair_id": "pair_8cfc8548abaac5b7", "doc_id": "doc_2c8f18fd479cd367", "variant": "D", "discipline": "biomedicine", "era": "2000-2009", "role": "data", "exact_match": 0.0, "edit_sim": 0.9294605809128631, "chrf": 90.3597295370775, "bleu": 82.4973983668462, "rougeL": 0.9324324324324325, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": 0.0} | ||||
| 10 | +{"pair_id": "pair_51ec58b1ca9af3cf", "doc_id": "doc_2c8f18fd479cd367", "variant": "E", "discipline": "biomedicine", "era": "2000-2009", "role": "data", "exact_match": 0.0, "edit_sim": 0.6021052631578947, "chrf": 91.11979667428962, "bleu": 79.06562336138828, "rougeL": 0.7837837837837838, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": 0.0} | ||||
| 11 | +{"pair_id": "pair_242060a529cbc25a", "doc_id": "doc_2c8f18fd479cd367", "variant": "E", "discipline": "biomedicine", "era": "2000-2009", "role": "data", "exact_match": 0.0, "edit_sim": 0.49250535331905787, "chrf": 91.05907086141916, "bleu": 76.07358601924324, "rougeL": 0.7027027027027027, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -1.0} | ||||
| 12 | +{"pair_id": "pair_ad5010e4d0261a40", "doc_id": "doc_2c8f18fd479cd367", "variant": "C", "discipline": "biomedicine", "era": "2000-2009", "role": "data", "exact_match": 0.0, "edit_sim": 0.9393305439330544, "chrf": 90.12761907257621, "bleu": 81.12815193983131, "rougeL": 0.9459459459459459, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -1.0} | ||||
| 13 | +{"pair_id": "pair_1376fcad6424f0fa", "doc_id": "doc_2c8f18fd479cd367", "variant": "A", "discipline": "biomedicine", "era": "2000-2009", "role": "discussion", "exact_match": 0.0, "edit_sim": 0.95, "chrf": 92.07369835049121, "bleu": 83.8021334375187, "rougeL": 0.9354838709677419, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": -1.0, "sentence_count_delta": 0.0} | ||||
| 14 | +{"pair_id": "pair_17fbf9f995f64446", "doc_id": "doc_2c8f18fd479cd367", "variant": "B", "discipline": "biomedicine", "era": "2000-2009", "role": "discussion", "exact_match": 0.0, "edit_sim": 0.9018264840182648, "chrf": 88.48653507762656, "bleu": 75.80522509169003, "rougeL": 0.8888888888888888, "len_ratio": 1.0327868852459017, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": -1.0, "sentence_count_delta": 0.0} | ||||
| 15 | +{"pair_id": "pair_fe62e92be3f24d0f", "doc_id": "doc_2c8f18fd479cd367", "variant": "C", "discipline": "biomedicine", "era": "2000-2009", "role": "discussion", "exact_match": 0.0, "edit_sim": 0.933806146572104, "chrf": 88.89851954431533, "bleu": 79.3876600216116, "rougeL": 0.9032258064516129, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": -1.0, "sentence_count_delta": 0.0} | ||||
| 16 | +{"pair_id": "pair_654d6afa29f53f45", "doc_id": "doc_2c8f18fd479cd367", "variant": "D", "discipline": "biomedicine", "era": "2000-2009", "role": "discussion", "exact_match": 0.0, "edit_sim": 0.933806146572104, "chrf": 88.89851954431533, "bleu": 79.3876600216116, "rougeL": 0.9032258064516129, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": -1.0, "sentence_count_delta": 0.0} | ||||
| 17 | +{"pair_id": "pair_d337266ac0a9ae36", "doc_id": "doc_2c8f18fd479cd367", "variant": "E", "discipline": "biomedicine", "era": "2000-2009", "role": "discussion", "exact_match": 0.0, "edit_sim": 0.933806146572104, "chrf": 88.89851954431533, "bleu": 79.3876600216116, "rougeL": 0.9032258064516129, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": -1.0, "sentence_count_delta": 0.0} | ||||
| 18 | +{"pair_id": "pair_faba89cbbf01184e", "doc_id": "doc_69d8fdb519110536", "variant": "A", "discipline": "biomedicine", "era": "2000-2009", "role": "abstract", "exact_match": 0.0, "edit_sim": 0.8637374860956618, "chrf": 91.50393745474274, "bleu": 81.66306276970191, "rougeL": 0.8867924528301887, "len_ratio": 1.0164609053497942, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -2.0} | ||||
| 19 | +{"pair_id": "pair_0044348fae33d9c9", "doc_id": "doc_69d8fdb519110536", "variant": "B", "discipline": "biomedicine", "era": "2000-2009", "role": "abstract", "exact_match": 0.0, "edit_sim": 0.9255555555555556, "chrf": 90.33005525761423, "bleu": 80.83591664955411, "rougeL": 0.9135338345864661, "len_ratio": 1.0246913580246915, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": 0.0} | ||||
| 20 | +{"pair_id": "pair_d14ee5ea783508ac", "doc_id": "doc_69d8fdb519110536", "variant": "D", "discipline": "biomedicine", "era": "2000-2009", "role": "abstract", "exact_match": 0.0, "edit_sim": 0.9298440979955457, "chrf": 90.73482306821884, "bleu": 81.77599525073695, "rougeL": 0.9172932330827067, "len_ratio": 1.0246913580246915, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": 0.0} | ||||
| 21 | +{"pair_id": "pair_d8907fc02c62eb15", "doc_id": "doc_69d8fdb519110536", "variant": "E", "discipline": "biomedicine", "era": "2000-2009", "role": "abstract", "exact_match": 0.0, "edit_sim": 0.7683333333333333, "chrf": 89.8551220932378, "bleu": 78.78084520430266, "rougeL": 0.819548872180451, "len_ratio": 1.0246913580246915, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -1.0} | ||||
| 22 | +{"pair_id": "pair_95528585a609d9f2", "doc_id": "doc_69d8fdb519110536", "variant": "E", "discipline": "biomedicine", "era": "2000-2009", "role": "abstract", "exact_match": 0.0, "edit_sim": 0.941638608305275, "chrf": 92.08136179954788, "bleu": 85.81645603310929, "rougeL": 0.9356060606060608, "len_ratio": 1.008230452674897, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": 0.0} | ||||
| 23 | +{"pair_id": "pair_edebb4e1082f8385", "doc_id": "doc_69d8fdb519110536", "variant": "C", "discipline": "biomedicine", "era": "2000-2009", "role": "abstract", "exact_match": 0.0, "edit_sim": 0.9306326304106548, "chrf": 90.4945773245641, "bleu": 80.47404367577207, "rougeL": 0.9172932330827067, "len_ratio": 1.0246913580246915, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -1.0} | ||||
| 24 | +{"pair_id": "pair_a260d6eb8785f314", "doc_id": "doc_c635c5544725270d", "variant": "A", "discipline": "biomedicine", "era": "2010-2017", "role": "data", "exact_match": 0.0, "edit_sim": 0.8801261829652997, "chrf": 95.70888407019373, "bleu": 90.97149213984494, "rougeL": 0.9230769230769231, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -1.0} | ||||
| 25 | +{"pair_id": "pair_e573ed74c8fb1b62", "doc_id": "doc_c635c5544725270d", "variant": "E", "discipline": "biomedicine", "era": "2010-2017", "role": "data", "exact_match": 0.0, "edit_sim": 0.8801261829652997, "chrf": 96.45724389071364, "bleu": 92.74154383234925, "rougeL": 0.9230769230769231, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": 0.0} | ||||
| 26 | +{"pair_id": "pair_e50c6c5ea350655f", "doc_id": "doc_c635c5544725270d", "variant": "E", "discipline": "biomedicine", "era": "2010-2017", "role": "data", "exact_match": 0.0, "edit_sim": 0.8035902851108765, "chrf": 96.35955814972328, "bleu": 92.30896631653356, "rougeL": 0.8601398601398601, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -1.0} | ||||
added
artifacts/metrics/vs_smoke/items_B1_generic_paraphrase.jsonl
+26 −0
@@ -0,0 +1,26 @@ | |||||
| 1 | +{"pair_id": "pair_f83b9af962e6f241", "doc_id": "doc_fec5ce1b5f4440a9", "variant": "A", "discipline": "biomedicine", "era": "2010-2017", "role": "related_work", "exact_match": 0.0, "edit_sim": 0.5748373101952278, "chrf": 86.88191499856877, "bleu": 75.14700449211313, "rougeL": 0.748201438848921, "len_ratio": 0.9857142857142858, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -1.0} | ||||
| 2 | +{"pair_id": "pair_e91a1b6883ac6e73", "doc_id": "doc_fec5ce1b5f4440a9", "variant": "B", "discipline": "biomedicine", "era": "2010-2017", "role": "related_work", "exact_match": 0.0, "edit_sim": 0.7960954446854664, "chrf": 89.24809455703415, "bleu": 80.66772189037918, "rougeL": 0.855072463768116, "len_ratio": 0.9714285714285714, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -1.0} | ||||
| 3 | +{"pair_id": "pair_a9af7ad1209bd3d3", "doc_id": "doc_fec5ce1b5f4440a9", "variant": "C", "discipline": "biomedicine", "era": "2010-2017", "role": "related_work", "exact_match": 0.0, "edit_sim": 0.9370932754880694, "chrf": 89.51649409867403, "bleu": 82.26760359337925, "rougeL": 0.920863309352518, "len_ratio": 0.9857142857142858, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -1.0} | ||||
| 4 | +{"pair_id": "pair_183ad7a4dd1ebebc", "doc_id": "doc_fec5ce1b5f4440a9", "variant": "D", "discipline": "biomedicine", "era": "2010-2017", "role": "related_work", "exact_match": 0.0, "edit_sim": 0.735357917570499, "chrf": 88.93380036665171, "bleu": 80.66730770098896, "rougeL": 0.7194244604316546, "len_ratio": 0.9857142857142858, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -1.0} | ||||
| 5 | +{"pair_id": "pair_bf9bdf18c8d57734", "doc_id": "doc_fec5ce1b5f4440a9", "variant": "E", "discipline": "biomedicine", "era": "2010-2017", "role": "related_work", "exact_match": 0.0, "edit_sim": 0.5748373101952278, "chrf": 86.88191499856877, "bleu": 75.14700449211313, "rougeL": 0.748201438848921, "len_ratio": 0.9857142857142858, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -1.0} | ||||
| 6 | +{"pair_id": "pair_13b7831e06faab4c", "doc_id": "doc_fec5ce1b5f4440a9", "variant": "E", "discipline": "biomedicine", "era": "2010-2017", "role": "related_work", "exact_match": 0.0, "edit_sim": 0.4099783080260304, "chrf": 90.7855536106871, "bleu": 84.51296529495087, "rougeL": 0.6376811594202899, "len_ratio": 0.9714285714285714, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": 0.0} | ||||
| 7 | +{"pair_id": "pair_008ecac459ef8ec7", "doc_id": "doc_2c8f18fd479cd367", "variant": "A", "discipline": "biomedicine", "era": "2000-2009", "role": "data", "exact_match": 0.0, "edit_sim": 0.7987551867219918, "chrf": 88.25752729300666, "bleu": 75.48784116332669, "rougeL": 0.8513513513513514, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -1.0} | ||||
| 8 | +{"pair_id": "pair_530abafa53fcf7d8", "doc_id": "doc_2c8f18fd479cd367", "variant": "B", "discipline": "biomedicine", "era": "2000-2009", "role": "data", "exact_match": 0.0, "edit_sim": 0.687374749498998, "chrf": 87.62114808326518, "bleu": 71.59086131059955, "rougeL": 0.6887417218543046, "len_ratio": 1.0434782608695652, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -1.0} | ||||
| 9 | +{"pair_id": "pair_8cfc8548abaac5b7", "doc_id": "doc_2c8f18fd479cd367", "variant": "D", "discipline": "biomedicine", "era": "2000-2009", "role": "data", "exact_match": 0.0, "edit_sim": 0.7261410788381742, "chrf": 89.41369136148526, "bleu": 78.95189327076916, "rougeL": 0.8378378378378378, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": 0.0} | ||||
| 10 | +{"pair_id": "pair_51ec58b1ca9af3cf", "doc_id": "doc_2c8f18fd479cd367", "variant": "E", "discipline": "biomedicine", "era": "2000-2009", "role": "data", "exact_match": 0.0, "edit_sim": 0.5995850622406639, "chrf": 87.94218144294332, "bleu": 74.25366032087813, "rougeL": 0.7567567567567567, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -1.0} | ||||
| 11 | +{"pair_id": "pair_242060a529cbc25a", "doc_id": "doc_2c8f18fd479cd367", "variant": "E", "discipline": "biomedicine", "era": "2000-2009", "role": "data", "exact_match": 0.0, "edit_sim": 0.48755186721991706, "chrf": 85.89441972919263, "bleu": 68.40195452018233, "rougeL": 0.6891891891891891, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -1.0} | ||||
| 12 | +{"pair_id": "pair_ad5010e4d0261a40", "doc_id": "doc_2c8f18fd479cd367", "variant": "C", "discipline": "biomedicine", "era": "2000-2009", "role": "data", "exact_match": 0.0, "edit_sim": 0.6120331950207469, "chrf": 86.23227687315284, "bleu": 70.22976462713258, "rougeL": 0.7702702702702703, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -1.0} | ||||
| 13 | +{"pair_id": "pair_1376fcad6424f0fa", "doc_id": "doc_2c8f18fd479cd367", "variant": "A", "discipline": "biomedicine", "era": "2000-2009", "role": "discussion", "exact_match": 0.0, "edit_sim": 0.933806146572104, "chrf": 88.89851954431533, "bleu": 79.3876600216116, "rougeL": 0.9032258064516129, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": -1.0, "sentence_count_delta": 0.0} | ||||
| 14 | +{"pair_id": "pair_17fbf9f995f64446", "doc_id": "doc_2c8f18fd479cd367", "variant": "B", "discipline": "biomedicine", "era": "2000-2009", "role": "discussion", "exact_match": 0.0, "edit_sim": 0.9018264840182648, "chrf": 88.48653507762656, "bleu": 75.80522509169003, "rougeL": 0.8888888888888888, "len_ratio": 1.0327868852459017, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": -1.0, "sentence_count_delta": 0.0} | ||||
| 15 | +{"pair_id": "pair_fe62e92be3f24d0f", "doc_id": "doc_2c8f18fd479cd367", "variant": "C", "discipline": "biomedicine", "era": "2000-2009", "role": "discussion", "exact_match": 0.0, "edit_sim": 0.933806146572104, "chrf": 88.89851954431533, "bleu": 79.3876600216116, "rougeL": 0.9032258064516129, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": -1.0, "sentence_count_delta": 0.0} | ||||
| 16 | +{"pair_id": "pair_654d6afa29f53f45", "doc_id": "doc_2c8f18fd479cd367", "variant": "D", "discipline": "biomedicine", "era": "2000-2009", "role": "discussion", "exact_match": 0.0, "edit_sim": 0.933806146572104, "chrf": 88.89851954431533, "bleu": 79.3876600216116, "rougeL": 0.9032258064516129, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": -1.0, "sentence_count_delta": 0.0} | ||||
| 17 | +{"pair_id": "pair_d337266ac0a9ae36", "doc_id": "doc_2c8f18fd479cd367", "variant": "E", "discipline": "biomedicine", "era": "2000-2009", "role": "discussion", "exact_match": 0.0, "edit_sim": 0.933806146572104, "chrf": 88.89851954431533, "bleu": 79.3876600216116, "rougeL": 0.9032258064516129, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": -1.0, "sentence_count_delta": 0.0} | ||||
| 18 | +{"pair_id": "pair_faba89cbbf01184e", "doc_id": "doc_69d8fdb519110536", "variant": "A", "discipline": "biomedicine", "era": "2000-2009", "role": "abstract", "exact_match": 0.0, "edit_sim": 0.8505298382598996, "chrf": 89.86383650042887, "bleu": 78.7982617935805, "rougeL": 0.8754716981132076, "len_ratio": 1.0164609053497942, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -2.0} | ||||
| 19 | +{"pair_id": "pair_0044348fae33d9c9", "doc_id": "doc_69d8fdb519110536", "variant": "B", "discipline": "biomedicine", "era": "2000-2009", "role": "abstract", "exact_match": 0.0, "edit_sim": 0.9255555555555556, "chrf": 90.33005525761423, "bleu": 80.83591664955411, "rougeL": 0.9135338345864661, "len_ratio": 1.0246913580246915, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": 0.0} | ||||
| 20 | +{"pair_id": "pair_d14ee5ea783508ac", "doc_id": "doc_69d8fdb519110536", "variant": "D", "discipline": "biomedicine", "era": "2000-2009", "role": "abstract", "exact_match": 0.0, "edit_sim": 0.76, "chrf": 90.11856583072493, "bleu": 79.93644685040046, "rougeL": 0.8458646616541353, "len_ratio": 1.0246913580246915, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": 0.0} | ||||
| 21 | +{"pair_id": "pair_d8907fc02c62eb15", "doc_id": "doc_69d8fdb519110536", "variant": "E", "discipline": "biomedicine", "era": "2000-2009", "role": "abstract", "exact_match": 0.0, "edit_sim": 0.7383592017738358, "chrf": 89.39412916067661, "bleu": 77.82679046102508, "rougeL": 0.8157894736842105, "len_ratio": 1.0246913580246915, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -1.0} | ||||
| 22 | +{"pair_id": "pair_95528585a609d9f2", "doc_id": "doc_69d8fdb519110536", "variant": "E", "discipline": "biomedicine", "era": "2000-2009", "role": "abstract", "exact_match": 0.0, "edit_sim": 0.7715565509518477, "chrf": 91.08603498597755, "bleu": 83.22048969785756, "rougeL": 0.8106060606060607, "len_ratio": 1.008230452674897, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": 0.0} | ||||
| 23 | +{"pair_id": "pair_edebb4e1082f8385", "doc_id": "doc_69d8fdb519110536", "variant": "C", "discipline": "biomedicine", "era": "2000-2009", "role": "abstract", "exact_match": 0.0, "edit_sim": 0.725, "chrf": 88.48058302353557, "bleu": 74.9922743405989, "rougeL": 0.763157894736842, "len_ratio": 1.0246913580246915, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -3.0} | ||||
| 24 | +{"pair_id": "pair_a260d6eb8785f314", "doc_id": "doc_c635c5544725270d", "variant": "A", "discipline": "biomedicine", "era": "2010-2017", "role": "data", "exact_match": 0.0, "edit_sim": 0.8801261829652997, "chrf": 95.70888407019373, "bleu": 90.97149213984494, "rougeL": 0.9230769230769231, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -1.0} | ||||
| 25 | +{"pair_id": "pair_e573ed74c8fb1b62", "doc_id": "doc_c635c5544725270d", "variant": "E", "discipline": "biomedicine", "era": "2010-2017", "role": "data", "exact_match": 0.0, "edit_sim": 0.8801261829652997, "chrf": 96.45724389071364, "bleu": 92.74154383234925, "rougeL": 0.9230769230769231, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": 0.0} | ||||
| 26 | +{"pair_id": "pair_e50c6c5ea350655f", "doc_id": "doc_c635c5544725270d", "variant": "E", "discipline": "biomedicine", "era": "2010-2017", "role": "data", "exact_match": 0.0, "edit_sim": 0.8035902851108765, "chrf": 96.35955814972328, "bleu": 92.30896631653356, "rougeL": 0.8601398601398601, "len_ratio": 1.0, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -1.0} | ||||
added
artifacts/metrics/vs_smoke/items_htr-m1-nano.jsonl
+26 −0
@@ -0,0 +1,26 @@ | |||||
| 1 | +{"pair_id": "pair_f83b9af962e6f241", "doc_id": "doc_fec5ce1b5f4440a9", "variant": "A", "discipline": "biomedicine", "era": "2010-2017", "role": "related_work", "exact_match": 0.0, "edit_sim": 0.2587859424920128, "chrf": 10.680191875644272, "bleu": 0.8213666792345169, "rougeL": 0.0881057268722467, "len_ratio": 2.242857142857143, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 0.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -3.0} | ||||
| 2 | +{"pair_id": "pair_e91a1b6883ac6e73", "doc_id": "doc_fec5ce1b5f4440a9", "variant": "B", "discipline": "biomedicine", "era": "2010-2017", "role": "related_work", "exact_match": 0.0, "edit_sim": 0.2777777777777778, "chrf": 11.307956372076665, "bleu": 0.67110567394756, "rougeL": 0.10045662100456622, "len_ratio": 2.1285714285714286, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 0.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -3.0} | ||||
| 3 | +{"pair_id": "pair_a9af7ad1209bd3d3", "doc_id": "doc_fec5ce1b5f4440a9", "variant": "C", "discipline": "biomedicine", "era": "2010-2017", "role": "related_work", "exact_match": 0.0, "edit_sim": 0.2678571428571429, "chrf": 11.340894021379004, "bleu": 0.682322356356094, "rougeL": 0.10185185185185185, "len_ratio": 2.085714285714286, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 0.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -3.0} | ||||
| 4 | +{"pair_id": "pair_183ad7a4dd1ebebc", "doc_id": "doc_fec5ce1b5f4440a9", "variant": "D", "discipline": "biomedicine", "era": "2010-2017", "role": "related_work", "exact_match": 0.0, "edit_sim": 0.27031249999999996, "chrf": 11.295796139569573, "bleu": 0.6602518153818534, "rougeL": 0.0990990990990991, "len_ratio": 2.1714285714285713, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 0.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -3.0} | ||||
| 5 | +{"pair_id": "pair_bf9bdf18c8d57734", "doc_id": "doc_fec5ce1b5f4440a9", "variant": "E", "discipline": "biomedicine", "era": "2010-2017", "role": "related_work", "exact_match": 0.0, "edit_sim": 0.27083333333333337, "chrf": 10.584130262330582, "bleu": 0.6210064664394688, "rougeL": 0.09523809523809523, "len_ratio": 2.3, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 0.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -3.0} | ||||
| 6 | +{"pair_id": "pair_13b7831e06faab4c", "doc_id": "doc_fec5ce1b5f4440a9", "variant": "E", "discipline": "biomedicine", "era": "2010-2017", "role": "related_work", "exact_match": 0.0, "edit_sim": 0.2727272727272727, "chrf": 11.303181285315683, "bleu": 0.6638305470715188, "rougeL": 0.09954751131221719, "len_ratio": 2.157142857142857, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 0.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -3.0} | ||||
| 7 | +{"pair_id": "pair_008ecac459ef8ec7", "doc_id": "doc_2c8f18fd479cd367", "variant": "A", "discipline": "biomedicine", "era": "2000-2009", "role": "data", "exact_match": 0.0, "edit_sim": 0.07096774193548383, "chrf": 1.4237062342590092, "bleu": 0.0, "rougeL": 0.0, "len_ratio": 0.014492753623188406, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 0.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -3.0} | ||||
| 8 | +{"pair_id": "pair_530abafa53fcf7d8", "doc_id": "doc_2c8f18fd479cd367", "variant": "B", "discipline": "biomedicine", "era": "2000-2009", "role": "data", "exact_match": 0.0, "edit_sim": 0.07096774193548383, "chrf": 1.4168369720816987, "bleu": 0.0, "rougeL": 0.0, "len_ratio": 0.014492753623188406, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 0.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -3.0} | ||||
| 9 | +{"pair_id": "pair_8cfc8548abaac5b7", "doc_id": "doc_2c8f18fd479cd367", "variant": "D", "discipline": "biomedicine", "era": "2000-2009", "role": "data", "exact_match": 0.0, "edit_sim": 0.07096774193548383, "chrf": 1.421409091068362, "bleu": 0.0, "rougeL": 0.0, "len_ratio": 0.014492753623188406, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 0.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -3.0} | ||||
| 10 | +{"pair_id": "pair_51ec58b1ca9af3cf", "doc_id": "doc_2c8f18fd479cd367", "variant": "E", "discipline": "biomedicine", "era": "2000-2009", "role": "data", "exact_match": 0.0, "edit_sim": 0.10322580645161294, "chrf": 3.097082767943988, "bleu": 0.00047182614506554476, "rougeL": 0.025, "len_ratio": 0.07246376811594203, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 0.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -3.0} | ||||
| 11 | +{"pair_id": "pair_242060a529cbc25a", "doc_id": "doc_2c8f18fd479cd367", "variant": "E", "discipline": "biomedicine", "era": "2000-2009", "role": "data", "exact_match": 0.0, "edit_sim": 0.22286661143330566, "chrf": 10.465719055728142, "bleu": 0.6292424664033717, "rougeL": 0.08408408408408409, "len_ratio": 3.739130434782609, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 0.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -3.0} | ||||
| 12 | +{"pair_id": "pair_ad5010e4d0261a40", "doc_id": "doc_2c8f18fd479cd367", "variant": "C", "discipline": "biomedicine", "era": "2000-2009", "role": "data", "exact_match": 0.0, "edit_sim": 0.07096774193548383, "chrf": 1.421409091068362, "bleu": 0.0, "rougeL": 0.0, "len_ratio": 0.014492753623188406, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 0.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -3.0} | ||||
| 13 | +{"pair_id": "pair_1376fcad6424f0fa", "doc_id": "doc_2c8f18fd479cd367", "variant": "A", "discipline": "biomedicine", "era": "2000-2009", "role": "discussion", "exact_match": 0.0, "edit_sim": 0.053012048192771055, "chrf": 1.0953981502575785, "bleu": 0.0, "rougeL": 0.0, "len_ratio": 0.01639344262295082, "placeholders_exact": 0.0, "placeholder_recall": 0.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": -3.0, "hedge_delta": -3.0, "sentence_count_delta": -2.0} | ||||
| 14 | +{"pair_id": "pair_17fbf9f995f64446", "doc_id": "doc_2c8f18fd479cd367", "variant": "B", "discipline": "biomedicine", "era": "2000-2009", "role": "discussion", "exact_match": 0.0, "edit_sim": 0.053012048192771055, "chrf": 1.0920486955339683, "bleu": 0.0, "rougeL": 0.0, "len_ratio": 0.01639344262295082, "placeholders_exact": 0.0, "placeholder_recall": 0.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": -3.0, "hedge_delta": -3.0, "sentence_count_delta": -2.0} | ||||
| 15 | +{"pair_id": "pair_fe62e92be3f24d0f", "doc_id": "doc_2c8f18fd479cd367", "variant": "C", "discipline": "biomedicine", "era": "2000-2009", "role": "discussion", "exact_match": 0.0, "edit_sim": 0.053012048192771055, "chrf": 1.0953981502575785, "bleu": 0.0, "rougeL": 0.0, "len_ratio": 0.01639344262295082, "placeholders_exact": 0.0, "placeholder_recall": 0.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": -3.0, "hedge_delta": -3.0, "sentence_count_delta": -2.0} | ||||
| 16 | +{"pair_id": "pair_654d6afa29f53f45", "doc_id": "doc_2c8f18fd479cd367", "variant": "D", "discipline": "biomedicine", "era": "2000-2009", "role": "discussion", "exact_match": 0.0, "edit_sim": 0.053012048192771055, "chrf": 1.0953981502575785, "bleu": 0.0, "rougeL": 0.0, "len_ratio": 0.01639344262295082, "placeholders_exact": 0.0, "placeholder_recall": 0.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": -3.0, "hedge_delta": -3.0, "sentence_count_delta": -2.0} | ||||
| 17 | +{"pair_id": "pair_d337266ac0a9ae36", "doc_id": "doc_2c8f18fd479cd367", "variant": "E", "discipline": "biomedicine", "era": "2000-2009", "role": "discussion", "exact_match": 0.0, "edit_sim": 0.053012048192771055, "chrf": 1.0953981502575785, "bleu": 0.0, "rougeL": 0.0, "len_ratio": 0.01639344262295082, "placeholders_exact": 0.0, "placeholder_recall": 0.0, "number_recall": 1.0, "number_precision": 1.0, "negation_delta": -3.0, "hedge_delta": -3.0, "sentence_count_delta": -2.0} | ||||
| 18 | +{"pair_id": "pair_faba89cbbf01184e", "doc_id": "doc_69d8fdb519110536", "variant": "A", "discipline": "biomedicine", "era": "2000-2009", "role": "abstract", "exact_match": 0.0, "edit_sim": 0.039887640449438155, "chrf": 0.8846909362721515, "bleu": 2.254669463411737e-38, "rougeL": 0.007490636704119851, "len_ratio": 0.012345679012345678, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 0.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": -1.0, "sentence_count_delta": -8.0} | ||||
| 19 | +{"pair_id": "pair_0044348fae33d9c9", "doc_id": "doc_69d8fdb519110536", "variant": "B", "discipline": "biomedicine", "era": "2000-2009", "role": "abstract", "exact_match": 0.0, "edit_sim": 0.039887640449438155, "chrf": 0.8846909362721515, "bleu": 2.254669463411737e-38, "rougeL": 0.007490636704119851, "len_ratio": 0.012345679012345678, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 0.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": -1.0, "sentence_count_delta": -8.0} | ||||
| 20 | +{"pair_id": "pair_d14ee5ea783508ac", "doc_id": "doc_69d8fdb519110536", "variant": "D", "discipline": "biomedicine", "era": "2000-2009", "role": "abstract", "exact_match": 0.0, "edit_sim": 0.039887640449438155, "chrf": 0.8846909362721515, "bleu": 2.254669463411737e-38, "rougeL": 0.007490636704119851, "len_ratio": 0.012345679012345678, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 0.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": -1.0, "sentence_count_delta": -8.0} | ||||
| 21 | +{"pair_id": "pair_d8907fc02c62eb15", "doc_id": "doc_69d8fdb519110536", "variant": "E", "discipline": "biomedicine", "era": "2000-2009", "role": "abstract", "exact_match": 0.0, "edit_sim": 0.039887640449438155, "chrf": 0.8846909362721515, "bleu": 2.254669463411737e-38, "rougeL": 0.007490636704119851, "len_ratio": 0.012345679012345678, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 0.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": -1.0, "sentence_count_delta": -8.0} | ||||
| 22 | +{"pair_id": "pair_95528585a609d9f2", "doc_id": "doc_69d8fdb519110536", "variant": "E", "discipline": "biomedicine", "era": "2000-2009", "role": "abstract", "exact_match": 0.0, "edit_sim": 0.039887640449438155, "chrf": 0.8846909362721515, "bleu": 2.254669463411737e-38, "rougeL": 0.007490636704119851, "len_ratio": 0.012345679012345678, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 0.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": -1.0, "sentence_count_delta": -8.0} | ||||
| 23 | +{"pair_id": "pair_edebb4e1082f8385", "doc_id": "doc_69d8fdb519110536", "variant": "C", "discipline": "biomedicine", "era": "2000-2009", "role": "abstract", "exact_match": 0.0, "edit_sim": 0.039887640449438155, "chrf": 0.8846909362721515, "bleu": 2.254669463411737e-38, "rougeL": 0.007490636704119851, "len_ratio": 0.012345679012345678, "placeholders_exact": 1.0, "placeholder_recall": 1.0, "number_recall": 0.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": -1.0, "sentence_count_delta": -8.0} | ||||
| 24 | +{"pair_id": "pair_a260d6eb8785f314", "doc_id": "doc_c635c5544725270d", "variant": "A", "discipline": "biomedicine", "era": "2010-2017", "role": "data", "exact_match": 0.0, "edit_sim": 0.13426423200859294, "chrf": 1.9870187606197238, "bleu": 0.22557991357122684, "rougeL": 0.004705882352941176, "len_ratio": 0.18439716312056736, "placeholders_exact": 0.0, "placeholder_recall": 0.5, "number_recall": 0.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -5.0} | ||||
| 25 | +{"pair_id": "pair_e573ed74c8fb1b62", "doc_id": "doc_c635c5544725270d", "variant": "E", "discipline": "biomedicine", "era": "2010-2017", "role": "data", "exact_match": 0.0, "edit_sim": 0.13748657357679916, "chrf": 1.9859664527694685, "bleu": 0.23683843384397701, "rougeL": 0.004705882352941176, "len_ratio": 0.19148936170212766, "placeholders_exact": 0.0, "placeholder_recall": 0.5, "number_recall": 0.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -5.0} | ||||
| 26 | +{"pair_id": "pair_e50c6c5ea350655f", "doc_id": "doc_c635c5544725270d", "variant": "E", "discipline": "biomedicine", "era": "2010-2017", "role": "data", "exact_match": 0.0, "edit_sim": 0.1407089151450054, "chrf": 1.9673351587825547, "bleu": 0.2435076258569401, "rougeL": 0.004705882352941176, "len_ratio": 0.19858156028368795, "placeholders_exact": 0.0, "placeholder_recall": 0.5, "number_recall": 0.0, "number_precision": 1.0, "negation_delta": 0.0, "hedge_delta": 0.0, "sentence_count_delta": -5.0} | ||||
added
artifacts/metrics/vs_smoke/report.json
+2114 −0
Large diff (2,114 lines) is not loaded by default to keep the page fast.Load diff
added
artifacts/metrics/vs_smoke/summary.md
+11 −0
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
added
artifacts/predictions/vs_smoke/outputs_B0_identity.jsonl
+26 −0
Large diff (26 lines) is not loaded by default to keep the page fast.Load diff
added
artifacts/predictions/vs_smoke/outputs_B1_generic_paraphrase.jsonl
+26 −0
Large diff (26 lines) is not loaded by default to keep the page fast.Load diff
added
artifacts/predictions/vs_smoke/outputs_htr-m1-nano.jsonl
+26 −0
Large diff (26 lines) is not loaded by default to keep the page fast.Load diff
added
artifacts/predictions/vs_smoke/predict_htr-m1-nano.json
+35 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added
artifacts/tokenizers/pilot_opus/tokenizer.model
+0 −0
Binary file not shown.
added
artifacts/tokenizers/pilot_opus/tokenizer.vocab
+16000 −0
Large diff (16,000 lines) is not loaded by default to keep the page fast.Load diff
added
artifacts/tokenizers/pilot_opus/tokenizer_config.json
+157 −0
Large diff (158 lines) is not loaded by default to keep the page fast.Load diff
added
artifacts/tokenizers/pilot_opus/tokenizer_train_summary.json
+33 −0
Large diff (33 lines) is not loaded by default to keep the page fast.Load diff
added
artifacts/tokenizers/vs_smoke/tokenizer.model
+0 −0
Binary file not shown.
added
artifacts/tokenizers/vs_smoke/tokenizer.vocab
+1385 −0
Large diff (1,385 lines) is not loaded by default to keep the page fast.Load diff
Diff truncated — file too large.