SPB Git
spboucher.ai

spb/wpai0002_uqo

Public

Release-candidate export smoke (clean load, CLI, demo), pilot-val figures/tables/numbers, STATUS

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Nov 9, 2026) parent a81f49d Browse files

20 changed files +560 −1

20 changed files
Unified Split
added artifacts/releases/HTR-Tiny-pilot-rc0-0.1.0/LICENSE-MODEL.txt +5 −0
@@ -0,0 +1,5 @@
1 +HTR model weights and code: Apache License 2.0 (see repository LICENSE).
2 +Training data: documents under CC0 / public domain / CC BY / CC BY-SA / CC BY-NC / CC BY-NC-SA licences (see the data card and
3 +docs/data_governance/LICENSE_MATRIX.md). Because CC BY-NC material may be among the training sources, the weights are released
4 +for NON-COMMERCIAL RESEARCH USE unless the release notes state that the checkpoint was trained on the CC0/CC BY/public-domain subset only.
5 +Outputs are synthetic reconstructions; they must not be presented as the original text or as evidence of human authorship.
added artifacts/releases/HTR-Tiny-pilot-rc0-0.1.0/MANIFEST.json +66 −0
@@ -0,0 +1,66 @@
1 +{
2 + "created_at": "2026-11-09T19:25:42Z",
3 + "git": {
4 + "commit": "a81f49dba436387f064488ab1f96645e83664e72",
5 + "branch": "main",
6 + "dirty": true
7 + },
8 + "libraries": {
9 + "python": "3.14.4",
10 + "torch": "2.12.0",
11 + "numpy": "2.5.4",
12 + "anthropic": "2.3.0",
13 + "transformers": "5.8.1",
14 + "sentencepiece": "0.2.2",
15 + "lxml": "6.1.3",
16 + "pyarrow": "24.0.0",
17 + "sacrebleu": "2.6.0",
18 + "datasketch": "2.0.0",
19 + "rapidfuzz": "3.14.6"
20 + },
21 + "hardware": {
22 + "platform": "macOS-27.0-arm64-arm-64bit-Mach-O",
23 + "machine": "arm64",
24 + "cpu_count": 18,
25 + "chip": "Apple M5 Max",
26 + "ram_gb": 48.0,
27 + "torch_mps": true,
28 + "torch_cuda": false
29 + },
30 + "release": "HTR-Tiny-pilot-rc0",
31 + "version": "0.1.0",
32 + "files": {
33 + "LICENSE-MODEL.txt": {
34 + "sha256": "cd4e864539cbcaf02c1a6ac91dda1eea02d9ccd51cb5fd7f108d45f179e57749",
35 + "bytes": 589
36 + },
37 + "README.md": {
38 + "sha256": "2a17a67265d96528627b1c15381deb33d3f8a27813284b166fbd721efc5975fd",
39 + "bytes": 4250
40 + },
41 + "config.json": {
42 + "sha256": "6ce3e9fae03ab0905f695140b8dcb1bc4ab11afab172efe7ab93941df0810169",
43 + "bytes": 447
44 + },
45 + "generation_config.json": {
46 + "sha256": "bbaf13a258fa07e5554fb97981d185f909cd85d055c669a724d91d8c68097cd7",
47 + "bytes": 275
48 + },
49 + "model.safetensors": {
50 + "sha256": "66e2bbfe94f4a38b8a383a070aafb5bae8edebf54c2c77db4a427a143ffa8882",
51 + "bytes": 109590452
52 + },
53 + "release_meta.json": {
54 + "sha256": "d81829c2cacf78e12269386f8df9f03321ff07579abcc812f63a0344e08fb8bd",
55 + "bytes": 2928
56 + },
57 + "tokenizer.model": {
58 + "sha256": "de8eac31c6407a86385bfe192f357ae528c16d8702c61bf83af9d69440255ec3",
59 + "bytes": 536604
60 + },
61 + "tokenizer_config.json": {
62 + "sha256": "bfaacbf864330aefcac2aee8180634a579fb2b902dc58158eabcb3121f7f6318",
63 + "bytes": 119
64 + }
65 + }
66 +}
added artifacts/releases/HTR-Tiny-pilot-rc0-0.1.0/README.md +118 −0
@@ -0,0 +1,118 @@
1 +---
2 +language: en
3 +license: apache-2.0
4 +library_name: htr
5 +tags: [text2text-generation, scientific-writing, style-reconstruction, research]
6 +pipeline_tag: text2text-generation
7 +---
8 +
9 +# HTR-Tiny-pilot-rc0 (v0.1.0)
10 +
11 +**Human Text Reconstruction Transformer** — a research model that maps an LLM-rewritten scientific passage back
12 +towards the distribution of the pre-2018 human-authored original. Outputs are **synthetic reconstructions**: they are
13 +not the original text, and they are not evidence of human authorship or of AI use. Part of UQO working paper WP-AI-0002.
14 +
15 +| Field | Value |
16 +|---|---|
17 +| Architecture | `m1` (encoder–decoder Transformer; m3 = dual-latent content/style with copy gate) |
18 +| Parameters | 27.4 M |
19 +| d_model / layers (enc+dec) | 384 / 4+4 |
20 +| Vocabulary (SentencePiece unigram) | 16000 |
21 +| Max sequence length | 640 tokens |
22 +| Style classes (m3) | n/a |
23 +| Training run | `pilot_m1_tiny_denoiseinit` |
24 +| Exported | 2026-11-09T19:25:42Z |
25 +
26 +## Intended use
27 +Research on how LLM rewriting transforms scientific prose; stylometric analysis; benchmarking inverse-paraphrase
28 +models; a tool for authors who want a draft re-aligned with historical scientific register **as a suggestion** that
29 +must be checked. Not intended for AI-text detection, authorship attribution, academic-integrity decisions, or
30 +production of text presented as human-written.
31 +
32 +## How to use
33 +```python
34 +from htr.inference.pipeline import HTRReconstructor
35 +rec = HTRReconstructor.from_pretrained("path/to/HTR-Tiny-pilot-rc0")
36 +r = rec.reconstruct("LLM-rewritten paragraph …", metadata={"discipline": "economics", "pub_year": 2012, "section_role": "results"})
37 +print(r.reconstruction); print(r.warnings)
38 +```
39 +CLI: `htr reconstruct --model path/to/HTR-Tiny-pilot-rc0 --text "…" --discipline economics --year 2012`.
40 +
41 +## Training data
42 +Pilot: 5 383 validated Claude Opus 5.5 rewrite pairs of 974 PMC Open Access passages (CC0/CC BY/CC BY-NC/CC BY-NC-SA/public domain), after denoising pretraining on 17 k train-split PMC passages; research candidate, NOT the release model
43 +
44 +## Evaluation (document-safe test split; identity baseline reported alongside)
45 +```json
46 +{
47 + "chrf": {
48 + "mean": 63.6045,
49 + "ci": [
50 + 62.1976,
51 + 64.861
52 + ]
53 + },
54 + "edit_sim": {
55 + "mean": 0.5777,
56 + "ci": [
57 + 0.5582,
58 + 0.5955
59 + ]
60 + },
61 + "rougeL": {
62 + "mean": 0.5996,
63 + "ci": [
64 + 0.5795,
65 + 0.6169
66 + ]
67 + },
68 + "exact_match": {
69 + "mean": 0.0,
70 + "ci": [
71 + 0.0,
72 + 0.0
73 + ]
74 + },
75 + "number_recall": {
76 + "mean": 0.9308,
77 + "ci": [
78 + 0.912,
79 + 0.9503
80 + ]
81 + },
82 + "placeholders_exact": {
83 + "mean": 0.8845,
84 + "ci": [
85 + 0.8303,
86 + 0.9381
87 + ]
88 + },
89 + "_baseline_B0_identity": {
90 + "chrf": 71.2394,
91 + "edit_sim": 0.6665,
92 + "rougeL": 0.6898,
93 + "exact_match": 0.0,
94 + "number_recall": 1.0,
95 + "placeholders_exact": 1.0
96 + }
97 +}
98 +```
99 +Metrics: chrF++, normalised edit similarity, ROUGE-L, exact match, number recall, placeholder (citation/math/URL)
100 +preservation, stylometric Wasserstein distance to originals. Confidence intervals are document-clustered bootstrap.
101 +
102 +## Limitations and risks
103 +- Reconstruction from a paraphrase is non-identifiable: many originals map to the same rewrite; the model returns one
104 + plausible estimate. Lexical overlap with the true original is bounded.
105 +- Factual integrity is enforced heuristically (protected spans, number checks); residual errors are flagged in
106 + `warnings`, not guaranteed absent. Always verify numbers, citations and negations.
107 +- Trained on Claude rewrites of CC-licensed biomedical, economics/finance, arXiv and public-domain book passages;
108 + behaviour on other rewriters, languages or genres is reported in the paper's generalisation experiments only.
109 +- Memorisation: training originals were deduplicated and the model is evaluated for nearest-neighbour overlap; see
110 + the paper. Do not use to recover specific copyrighted texts.
111 +
112 +## Licence
113 +Code and weights: Apache-2.0. Data licences: see `LICENSE-MODEL.txt` and the data card. Attribution to NLM/PMC,
114 +arXiv, publishers and Project Gutenberg sources is in the data card.
115 +
116 +## Citation
117 +Boucher, S.-P. (2026). *Human Text Reconstruction: Learning the Inverse of LLM-Induced Transformations in Scientific
118 +Writing.* UQO Working Paper AI-0002.
added artifacts/releases/HTR-Tiny-pilot-rc0-0.1.0/config.json +26 −0
@@ -0,0 +1,26 @@
1 +{
2 + "model_type": "htr",
3 + "arch": "m1",
4 + "vocab_size": 16000,
5 + "d_model": 384,
6 + "n_heads": 6,
7 + "d_ff": 1536,
8 + "n_enc_layers": 4,
9 + "n_dec_layers": 4,
10 + "dropout": 0.1,
11 + "max_len": 640,
12 + "tie_embeddings": true,
13 + "pos": "rope",
14 + "n_disc": 16,
15 + "n_era": 9,
16 + "n_role": 16,
17 + "n_source": 9,
18 + "n_styles": 0,
19 + "style_tokens": 4,
20 + "grl_lambda": 0.1,
21 + "use_copy": true,
22 + "pad_id": 0,
23 + "bos_id": 1,
24 + "eos_id": 2,
25 + "n_params": 27394945
26 +}
added artifacts/releases/HTR-Tiny-pilot-rc0-0.1.0/generation_config.json +7 −0
@@ -0,0 +1,7 @@
1 +{
2 + "decoding": "greedy",
3 + "beam": 1,
4 + "max_new_tokens": "1.6 x input tokens + 16 (capped by config.max_len)",
5 + "deterministic": true,
6 + "placeholder_protocol": "citations/math/urls are replaced by ⟦Cn⟧/⟦Mn⟧/⟦Un⟧ before encoding and restored after decoding"
7 +}
added artifacts/releases/HTR-Tiny-pilot-rc0-0.1.0/model.safetensors +0 −0

Binary file not shown.

added artifacts/releases/HTR-Tiny-pilot-rc0-0.1.0/release_meta.json +134 −0
@@ -0,0 +1,134 @@
1 +{
2 + "name": "HTR-Tiny-pilot-rc0",
3 + "text_meta_prefix": false,
4 + "style_classes": null,
5 + "run": "pilot_m1_tiny_denoiseinit",
6 + "version": "0.1.0",
7 + "exported_at": "2026-11-09T19:25:42Z",
8 + "arch": "m1",
9 + "n_params": 27394945,
10 + "source_checkpoint": "artifacts/runs/pilot_m1_tiny_denoiseinit/best",
11 + "checkpoint_meta": {
12 + "created_at": "2026-11-09T19:08:19Z",
13 + "git": {
14 + "commit": null,
15 + "branch": null,
16 + "dirty": false
17 + },
18 + "libraries": {
19 + "python": "3.14.5",
20 + "torch": "2.15.0",
21 + "numpy": "2.5.4",
22 + "anthropic": null,
23 + "transformers": null,
24 + "sentencepiece": "0.2.3",
25 + "lxml": "6.1.3",
26 + "pyarrow": "26.0.1",
27 + "sacrebleu": "2.6.0",
28 + "datasketch": "2.0.0",
29 + "rapidfuzz": "3.14.6"
30 + },
31 + "hardware": {
32 + "platform": "macOS-26.6-arm64-arm-64bit-Mach-O",
33 + "machine": "arm64",
34 + "cpu_count": 28,
35 + "chip": "Apple M3 Ultra",
36 + "ram_gb": 96.0,
37 + "torch_mps": true,
38 + "torch_cuda": false
39 + },
40 + "step": 250,
41 + "tokens_seen": 2152249,
42 + "accel_seconds": 45.8,
43 + "train_config": {
44 + "lr": 0.0002,
45 + "weight_decay": 0.01,
46 + "betas": [
47 + 0.9,
48 + 0.98
49 + ],
50 + "warmup_steps": 100,
51 + "max_steps": 4000,
52 + "accum": 1,
53 + "clip": 1.0,
54 + "eval_every": 250,
55 + "log_every": 25,
56 + "patience": 6,
57 + "amp": "auto",
58 + "seed": 1,
59 + "max_tokens": 6144,
60 + "max_batch": 48,
61 + "loss": {
62 + "ce": 1.0,
63 + "fact": 0.0,
64 + "style": 0.0,
65 + "adv": 0.0,
66 + "con": 0.0,
67 + "sem": 0.0,
68 + "orth": 0.0,
69 + "label_smoothing": 0.1,
70 + "temperature": 0.07
71 + },
72 + "device": "auto",
73 + "keep_last": 2
74 + },
75 + "eval": {
76 + "val_ce": 1.6078313916623073,
77 + "val_ppl": 4.99197384495966,
78 + "val_tokens": 119891
79 + }
80 + },
81 + "metrics": {
82 + "chrf": {
83 + "mean": 63.6045,
84 + "ci": [
85 + 62.1976,
86 + 64.861
87 + ]
88 + },
89 + "edit_sim": {
90 + "mean": 0.5777,
91 + "ci": [
92 + 0.5582,
93 + 0.5955
94 + ]
95 + },
96 + "rougeL": {
97 + "mean": 0.5996,
98 + "ci": [
99 + 0.5795,
100 + 0.6169
101 + ]
102 + },
103 + "exact_match": {
104 + "mean": 0.0,
105 + "ci": [
106 + 0.0,
107 + 0.0
108 + ]
109 + },
110 + "number_recall": {
111 + "mean": 0.9308,
112 + "ci": [
113 + 0.912,
114 + 0.9503
115 + ]
116 + },
117 + "placeholders_exact": {
118 + "mean": 0.8845,
119 + "ci": [
120 + 0.8303,
121 + 0.9381
122 + ]
123 + },
124 + "_baseline_B0_identity": {
125 + "chrf": 71.2394,
126 + "edit_sim": 0.6665,
127 + "rougeL": 0.6898,
128 + "exact_match": 0.0,
129 + "number_recall": 1.0,
130 + "placeholders_exact": 1.0
131 + }
132 + },
133 + "training_data": "Pilot: 5 383 validated Claude Opus 5.5 rewrite pairs of 974 PMC Open Access passages (CC0/CC BY/CC BY-NC/CC BY-NC-SA/public domain), after denoising pretraining on 17 k train-split PMC passages; research candidate, NOT the release model"
134 +}
added artifacts/releases/HTR-Tiny-pilot-rc0-0.1.0/tokenizer.model +0 −0

Binary file not shown.

added artifacts/releases/HTR-Tiny-pilot-rc0-0.1.0/tokenizer_config.json +8 −0
@@ -0,0 +1,8 @@
1 +{
2 + "type": "sentencepiece",
3 + "file": "tokenizer.model",
4 + "pad_id": 0,
5 + "bos_id": 1,
6 + "eos_id": 2,
7 + "unk_id": 3
8 +}
modified docs/STATUS.md +1 −1
@@ -8,7 +8,7 @@
8 8 | C — leakage-safe splits, reproducible baselines, benchmark + statistical plan | **GO** (splits frozen; B0/B1 run on val; pre-registration v1.0 frozen at `e74b7126` + A1) | `docs/gates/GATE_C.md` |
9 9 | D — M1/M2/M3 training, one-batch overfit, pilot results | in progress — exploratory val (502 pairs): B0 chrF 71.2 · B1 68.4 · **B2 T5-small 68.8** (W1 to originals 0.112 < 0.170, i.e. closer style distribution but lower lexical fidelity) · **M1-tiny from scratch 32.9** (degenerate). Best val CE over seeds 1–3: M1 3.69/4.09/4.31 · M2 3.72/5.03/3.71 · M3 token-only 3.80/3.74/3.58 · M3 with auxiliary losses ≈ 5.4 → diagnosed (D18): the L_fact normalisation, not the architecture; fixed and re-running. **Denoising pretraining (D17)** works: pilot denoiser (12 000 steps on 17 k train passages) val CE 1.58; fine-tuned on the pilot pairs → **val chrF 63.6** [62.2, 64.9] (from scratch 32.9), number recall 0.93, placeholders 0.88, W1 0.150 (< identity 0.170); still −7.6 chrF below identity on this small pair set (B2 T5-small −2.4). Denoiser alone (no pair fine-tuning) chrF 58.1. Medium denoisers tiny (M3U96b) and small 130 M (M4M64a) running, fine-tuning queued after them. | `artifacts/metrics/pilot_val_exploratory/`, `artifacts/runs/`, `docs/DECISION_LOG.md` D17–D18 |
10 10 | E — larger training | not started (depends on D) | — |
11 −| F — release + paper | scaffolding only (release exporter, CLI, demo, model-card template, paper skeleton compiles: 5 pages of placeholders) | `src/htr/release`, `src/htr/cli.py`, `demo/`, `paper/` |
11 +| F — release + paper | tooling validated end to end on the pilot denoise-initialised model: `artifacts/releases/HTR-Tiny-pilot-rc0-0.1.0` (SafeTensors 105 MB, tokenizer, config, model card, MANIFEST with sha256, LICENSE note) loads in a clean `python -I` interpreter, the `htr` CLI reconstructs, the web demo answers `/health` and `/api/reconstruct`. Output quality is pilot-level (not a release). Paper: 46 pages, non-results sections drafted, results/ablations/robustness/human-eval/threats/limitations/conclusion pending measured results. | `scripts/export_release.py`, `src/htr/cli.py`, `demo/`, `paper/` |
12 12
13 13 ## API spend (ledgers, Message Batches = 50 % price)
14 14 | Job | Model | Requests | ok / refused | USD |
modified paper/figures/FIGURE_PROVENANCE.md +1 −0
@@ -1,3 +1,4 @@
1 1 # Figure provenance
2 2
3 3 - Smoke: generated from `artifacts/metrics/vs_smoke/report.json` (created 2026-11-09T17:33:43Z, git 43899c8924172a93eb7fdb44567c22c0c6bb1e47) and runs ['artifacts/runs/vs_smoke-m1-nano']
4 +- PilotVal: generated from `artifacts/metrics/pilot_val_exploratory/report.json` (created 2026-11-09T19:23:50Z, git c310982a0a448cfba546579ee1883679f6565058) and runs ['artifacts/runs/pilot_m1_tiny', 'artifacts/runs/pilot_m2_tiny', 'artifacts/runs/pilot_m1_tiny_denoiseinit']
added paper/figures/fig_by_discipline_pilotval.pdf +0 −0

Binary file not shown.

added paper/figures/fig_by_era_pilotval.pdf +0 −0

Binary file not shown.

added paper/figures/fig_by_role_pilotval.pdf +0 −0

Binary file not shown.

added paper/figures/fig_by_variant_pilotval.pdf +0 −0

Binary file not shown.

added paper/figures/fig_overall_pilotval.pdf +0 −0

Binary file not shown.

added paper/figures/fig_stylometry_pilotval.pdf +0 −0

Binary file not shown.

added paper/figures/fig_training_pilotval.pdf +0 −0

Binary file not shown.

added paper/numbers_pilotval.tex +182 −0
@@ -0,0 +1,182 @@
1 +% generated by scripts/build_figures.py from PilotVal — do not edit
2 +\newcommand{\PilotValBIdentityChrf}{71.24}
3 +\newcommand{\PilotValBIdentityChrfLo}{69.69}
4 +\newcommand{\PilotValBIdentityChrfHi}{72.65}
5 +\newcommand{\PilotValBIdentityEditSim}{0.666}
6 +\newcommand{\PilotValBIdentityEditSimLo}{0.645}
7 +\newcommand{\PilotValBIdentityEditSimHi}{0.686}
8 +\newcommand{\PilotValBIdentityRougel}{0.690}
9 +\newcommand{\PilotValBIdentityRougelLo}{0.670}
10 +\newcommand{\PilotValBIdentityRougelHi}{0.708}
11 +\newcommand{\PilotValBIdentityExactMatch}{0.000}
12 +\newcommand{\PilotValBIdentityExactMatchLo}{0.000}
13 +\newcommand{\PilotValBIdentityExactMatchHi}{0.000}
14 +\newcommand{\PilotValBIdentityNumberRecall}{1.000}
15 +\newcommand{\PilotValBIdentityNumberRecallLo}{1.000}
16 +\newcommand{\PilotValBIdentityNumberRecallHi}{1.000}
17 +\newcommand{\PilotValBIdentityPlaceholdersExact}{1.000}
18 +\newcommand{\PilotValBIdentityPlaceholdersExactLo}{1.000}
19 +\newcommand{\PilotValBIdentityPlaceholdersExactHi}{1.000}
20 +\newcommand{\PilotValBIdentityN}{502}
21 +\newcommand{\PilotValBGenericParaphraseChrf}{68.41}
22 +\newcommand{\PilotValBGenericParaphraseChrfLo}{66.90}
23 +\newcommand{\PilotValBGenericParaphraseChrfHi}{69.99}
24 +\newcommand{\PilotValBGenericParaphraseEditSim}{0.561}
25 +\newcommand{\PilotValBGenericParaphraseEditSimLo}{0.545}
26 +\newcommand{\PilotValBGenericParaphraseEditSimHi}{0.579}
27 +\newcommand{\PilotValBGenericParaphraseRougel}{0.612}
28 +\newcommand{\PilotValBGenericParaphraseRougelLo}{0.593}
29 +\newcommand{\PilotValBGenericParaphraseRougelHi}{0.632}
30 +\newcommand{\PilotValBGenericParaphraseExactMatch}{0.000}
31 +\newcommand{\PilotValBGenericParaphraseExactMatchLo}{0.000}
32 +\newcommand{\PilotValBGenericParaphraseExactMatchHi}{0.000}
33 +\newcommand{\PilotValBGenericParaphraseNumberRecall}{1.000}
34 +\newcommand{\PilotValBGenericParaphraseNumberRecallLo}{1.000}
35 +\newcommand{\PilotValBGenericParaphraseNumberRecallHi}{1.000}
36 +\newcommand{\PilotValBGenericParaphrasePlaceholdersExact}{1.000}
37 +\newcommand{\PilotValBGenericParaphrasePlaceholdersExactLo}{1.000}
38 +\newcommand{\PilotValBGenericParaphrasePlaceholdersExactHi}{1.000}
39 +\newcommand{\PilotValBGenericParaphraseN}{502}
40 +\newcommand{\PilotValBTSmallChrf}{68.85}
41 +\newcommand{\PilotValBTSmallChrfLo}{67.66}
42 +\newcommand{\PilotValBTSmallChrfHi}{69.94}
43 +\newcommand{\PilotValBTSmallEditSim}{0.638}
44 +\newcommand{\PilotValBTSmallEditSimLo}{0.621}
45 +\newcommand{\PilotValBTSmallEditSimHi}{0.657}
46 +\newcommand{\PilotValBTSmallRougel}{0.671}
47 +\newcommand{\PilotValBTSmallRougelLo}{0.654}
48 +\newcommand{\PilotValBTSmallRougelHi}{0.687}
49 +\newcommand{\PilotValBTSmallExactMatch}{0.000}
50 +\newcommand{\PilotValBTSmallExactMatchLo}{0.000}
51 +\newcommand{\PilotValBTSmallExactMatchHi}{0.000}
52 +\newcommand{\PilotValBTSmallNumberRecall}{0.945}
53 +\newcommand{\PilotValBTSmallNumberRecallLo}{0.919}
54 +\newcommand{\PilotValBTSmallNumberRecallHi}{0.969}
55 +\newcommand{\PilotValBTSmallPlaceholdersExact}{0.691}
56 +\newcommand{\PilotValBTSmallPlaceholdersExactLo}{0.585}
57 +\newcommand{\PilotValBTSmallPlaceholdersExactHi}{0.787}
58 +\newcommand{\PilotValBTSmallN}{502}
59 +\newcommand{\PilotValHtrDenoiseOnlyChrf}{58.06}
60 +\newcommand{\PilotValHtrDenoiseOnlyChrfLo}{55.21}
61 +\newcommand{\PilotValHtrDenoiseOnlyChrfHi}{60.87}
62 +\newcommand{\PilotValHtrDenoiseOnlyEditSim}{0.456}
63 +\newcommand{\PilotValHtrDenoiseOnlyEditSimLo}{0.421}
64 +\newcommand{\PilotValHtrDenoiseOnlyEditSimHi}{0.491}
65 +\newcommand{\PilotValHtrDenoiseOnlyRougel}{0.493}
66 +\newcommand{\PilotValHtrDenoiseOnlyRougelLo}{0.459}
67 +\newcommand{\PilotValHtrDenoiseOnlyRougelHi}{0.526}
68 +\newcommand{\PilotValHtrDenoiseOnlyExactMatch}{0.000}
69 +\newcommand{\PilotValHtrDenoiseOnlyExactMatchLo}{0.000}
70 +\newcommand{\PilotValHtrDenoiseOnlyExactMatchHi}{0.000}
71 +\newcommand{\PilotValHtrDenoiseOnlyNumberRecall}{0.856}
72 +\newcommand{\PilotValHtrDenoiseOnlyNumberRecallLo}{0.806}
73 +\newcommand{\PilotValHtrDenoiseOnlyNumberRecallHi}{0.902}
74 +\newcommand{\PilotValHtrDenoiseOnlyPlaceholdersExact}{0.741}
75 +\newcommand{\PilotValHtrDenoiseOnlyPlaceholdersExactLo}{0.649}
76 +\newcommand{\PilotValHtrDenoiseOnlyPlaceholdersExactHi}{0.824}
77 +\newcommand{\PilotValHtrDenoiseOnlyN}{502}
78 +\newcommand{\PilotValHtrMTinyDenoiseinitChrf}{63.60}
79 +\newcommand{\PilotValHtrMTinyDenoiseinitChrfLo}{62.20}
80 +\newcommand{\PilotValHtrMTinyDenoiseinitChrfHi}{64.86}
81 +\newcommand{\PilotValHtrMTinyDenoiseinitEditSim}{0.578}
82 +\newcommand{\PilotValHtrMTinyDenoiseinitEditSimLo}{0.558}
83 +\newcommand{\PilotValHtrMTinyDenoiseinitEditSimHi}{0.595}
84 +\newcommand{\PilotValHtrMTinyDenoiseinitRougel}{0.600}
85 +\newcommand{\PilotValHtrMTinyDenoiseinitRougelLo}{0.579}
86 +\newcommand{\PilotValHtrMTinyDenoiseinitRougelHi}{0.617}
87 +\newcommand{\PilotValHtrMTinyDenoiseinitExactMatch}{0.000}
88 +\newcommand{\PilotValHtrMTinyDenoiseinitExactMatchLo}{0.000}
89 +\newcommand{\PilotValHtrMTinyDenoiseinitExactMatchHi}{0.000}
90 +\newcommand{\PilotValHtrMTinyDenoiseinitNumberRecall}{0.931}
91 +\newcommand{\PilotValHtrMTinyDenoiseinitNumberRecallLo}{0.912}
92 +\newcommand{\PilotValHtrMTinyDenoiseinitNumberRecallHi}{0.950}
93 +\newcommand{\PilotValHtrMTinyDenoiseinitPlaceholdersExact}{0.884}
94 +\newcommand{\PilotValHtrMTinyDenoiseinitPlaceholdersExactLo}{0.830}
95 +\newcommand{\PilotValHtrMTinyDenoiseinitPlaceholdersExactHi}{0.938}
96 +\newcommand{\PilotValHtrMTinyDenoiseinitN}{502}
97 +\newcommand{\PilotValHtrMTinyChrf}{32.86}
98 +\newcommand{\PilotValHtrMTinyChrfLo}{30.87}
99 +\newcommand{\PilotValHtrMTinyChrfHi}{34.47}
100 +\newcommand{\PilotValHtrMTinyEditSim}{0.287}
101 +\newcommand{\PilotValHtrMTinyEditSimLo}{0.274}
102 +\newcommand{\PilotValHtrMTinyEditSimHi}{0.299}
103 +\newcommand{\PilotValHtrMTinyRougel}{0.241}
104 +\newcommand{\PilotValHtrMTinyRougelLo}{0.224}
105 +\newcommand{\PilotValHtrMTinyRougelHi}{0.255}
106 +\newcommand{\PilotValHtrMTinyExactMatch}{0.000}
107 +\newcommand{\PilotValHtrMTinyExactMatchLo}{0.000}
108 +\newcommand{\PilotValHtrMTinyExactMatchHi}{0.000}
109 +\newcommand{\PilotValHtrMTinyNumberRecall}{0.571}
110 +\newcommand{\PilotValHtrMTinyNumberRecallLo}{0.503}
111 +\newcommand{\PilotValHtrMTinyNumberRecallHi}{0.659}
112 +\newcommand{\PilotValHtrMTinyPlaceholdersExact}{0.633}
113 +\newcommand{\PilotValHtrMTinyPlaceholdersExactLo}{0.505}
114 +\newcommand{\PilotValHtrMTinyPlaceholdersExactHi}{0.742}
115 +\newcommand{\PilotValHtrMTinyN}{502}
116 +\newcommand{\PilotValBGenericParaphraseChrfDiff}{-2.82}
117 +\newcommand{\PilotValBGenericParaphraseChrfPHolm}{0.000}
118 +\newcommand{\PilotValBGenericParaphraseEditSimDiff}{-0.105}
119 +\newcommand{\PilotValBGenericParaphraseEditSimPHolm}{0.000}
120 +\newcommand{\PilotValBGenericParaphraseRougelDiff}{-0.078}
121 +\newcommand{\PilotValBGenericParaphraseRougelPHolm}{0.000}
122 +\newcommand{\PilotValBGenericParaphraseExactMatchDiff}{+0.000}
123 +\newcommand{\PilotValBGenericParaphraseExactMatchPHolm}{0.000}
124 +\newcommand{\PilotValBGenericParaphraseNumberRecallDiff}{+0.000}
125 +\newcommand{\PilotValBGenericParaphraseNumberRecallPHolm}{0.000}
126 +\newcommand{\PilotValBGenericParaphrasePlaceholdersExactDiff}{+0.000}
127 +\newcommand{\PilotValBGenericParaphrasePlaceholdersExactPHolm}{0.000}
128 +\newcommand{\PilotValBTSmallChrfDiff}{-2.39}
129 +\newcommand{\PilotValBTSmallChrfPHolm}{0.000}
130 +\newcommand{\PilotValBTSmallEditSimDiff}{-0.028}
131 +\newcommand{\PilotValBTSmallEditSimPHolm}{0.000}
132 +\newcommand{\PilotValBTSmallRougelDiff}{-0.019}
133 +\newcommand{\PilotValBTSmallRougelPHolm}{0.000}
134 +\newcommand{\PilotValBTSmallExactMatchDiff}{+0.000}
135 +\newcommand{\PilotValBTSmallExactMatchPHolm}{0.000}
136 +\newcommand{\PilotValBTSmallNumberRecallDiff}{-0.055}
137 +\newcommand{\PilotValBTSmallNumberRecallPHolm}{0.000}
138 +\newcommand{\PilotValBTSmallPlaceholdersExactDiff}{-0.309}
139 +\newcommand{\PilotValBTSmallPlaceholdersExactPHolm}{0.000}
140 +\newcommand{\PilotValHtrDenoiseOnlyChrfDiff}{-13.18}
141 +\newcommand{\PilotValHtrDenoiseOnlyChrfPHolm}{0.000}
142 +\newcommand{\PilotValHtrDenoiseOnlyEditSimDiff}{-0.210}
143 +\newcommand{\PilotValHtrDenoiseOnlyEditSimPHolm}{0.000}
144 +\newcommand{\PilotValHtrDenoiseOnlyRougelDiff}{-0.196}
145 +\newcommand{\PilotValHtrDenoiseOnlyRougelPHolm}{0.000}
146 +\newcommand{\PilotValHtrDenoiseOnlyExactMatchDiff}{+0.000}
147 +\newcommand{\PilotValHtrDenoiseOnlyExactMatchPHolm}{0.000}
148 +\newcommand{\PilotValHtrDenoiseOnlyNumberRecallDiff}{-0.144}
149 +\newcommand{\PilotValHtrDenoiseOnlyNumberRecallPHolm}{0.000}
150 +\newcommand{\PilotValHtrDenoiseOnlyPlaceholdersExactDiff}{-0.259}
151 +\newcommand{\PilotValHtrDenoiseOnlyPlaceholdersExactPHolm}{0.000}
152 +\newcommand{\PilotValHtrMTinyDenoiseinitChrfDiff}{-7.63}
153 +\newcommand{\PilotValHtrMTinyDenoiseinitChrfPHolm}{0.000}
154 +\newcommand{\PilotValHtrMTinyDenoiseinitEditSimDiff}{-0.089}
155 +\newcommand{\PilotValHtrMTinyDenoiseinitEditSimPHolm}{0.000}
156 +\newcommand{\PilotValHtrMTinyDenoiseinitRougelDiff}{-0.090}
157 +\newcommand{\PilotValHtrMTinyDenoiseinitRougelPHolm}{0.000}
158 +\newcommand{\PilotValHtrMTinyDenoiseinitExactMatchDiff}{+0.000}
159 +\newcommand{\PilotValHtrMTinyDenoiseinitExactMatchPHolm}{0.000}
160 +\newcommand{\PilotValHtrMTinyDenoiseinitNumberRecallDiff}{-0.069}
161 +\newcommand{\PilotValHtrMTinyDenoiseinitNumberRecallPHolm}{0.000}
162 +\newcommand{\PilotValHtrMTinyDenoiseinitPlaceholdersExactDiff}{-0.116}
163 +\newcommand{\PilotValHtrMTinyDenoiseinitPlaceholdersExactPHolm}{0.000}
164 +\newcommand{\PilotValHtrMTinyChrfDiff}{-38.38}
165 +\newcommand{\PilotValHtrMTinyChrfPHolm}{0.000}
166 +\newcommand{\PilotValHtrMTinyEditSimDiff}{-0.379}
167 +\newcommand{\PilotValHtrMTinyEditSimPHolm}{0.000}
168 +\newcommand{\PilotValHtrMTinyRougelDiff}{-0.449}
169 +\newcommand{\PilotValHtrMTinyRougelPHolm}{0.000}
170 +\newcommand{\PilotValHtrMTinyExactMatchDiff}{+0.000}
171 +\newcommand{\PilotValHtrMTinyExactMatchPHolm}{0.000}
172 +\newcommand{\PilotValHtrMTinyNumberRecallDiff}{-0.429}
173 +\newcommand{\PilotValHtrMTinyNumberRecallPHolm}{0.000}
174 +\newcommand{\PilotValHtrMTinyPlaceholdersExactDiff}{-0.367}
175 +\newcommand{\PilotValHtrMTinyPlaceholdersExactPHolm}{0.000}
176 +\newcommand{\PilotValWRewrite}{0.170}
177 +\newcommand{\PilotValWBIdentity}{0.170}
178 +\newcommand{\PilotValWBGenericParaphrase}{0.281}
179 +\newcommand{\PilotValWBTSmall}{0.112}
180 +\newcommand{\PilotValWHtrDenoiseOnly}{0.215}
181 +\newcommand{\PilotValWHtrMTinyDenoiseinit}{0.150}
182 +\newcommand{\PilotValWHtrMTiny}{1.699}
added paper/tables/tab_overall_pilotval.tex +12 −0
@@ -0,0 +1,12 @@
1 +\begin{tabular}{l r c c c c c}
2 +\toprule
3 +System & $n$ & chrF++ & edit similarity & ROUGE-L & number recall & placeholders exact \\
4 +\midrule
5 +B0 identity & 502 & 71.24 [69.69, 72.65] & 0.666 [0.645, 0.686] & 0.690 [0.670, 0.708] & 1.000 [1.000, 1.000] & 1.000 [1.000, 1.000] \\
6 +B1 paraphrase & 502 & 68.41 [66.90, 69.99] & 0.561 [0.545, 0.579] & 0.612 [0.593, 0.632] & 1.000 [1.000, 1.000] & 1.000 [1.000, 1.000] \\
7 +B2 t5-small & 502 & 68.85 [67.66, 69.94] & 0.638 [0.621, 0.657] & 0.671 [0.654, 0.687] & 0.945 [0.919, 0.969] & 0.691 [0.585, 0.787] \\
8 +denoise-only & 502 & 58.06 [55.21, 60.87] & 0.456 [0.421, 0.491] & 0.493 [0.459, 0.526] & 0.856 [0.806, 0.902] & 0.741 [0.649, 0.824] \\
9 +m1-tiny-denoiseinit & 502 & 63.60 [62.20, 64.86] & 0.578 [0.558, 0.595] & 0.600 [0.579, 0.617] & 0.931 [0.912, 0.950] & 0.884 [0.830, 0.938] \\
10 +m1-tiny & 502 & 32.86 [30.87, 34.47] & 0.287 [0.274, 0.299] & 0.241 [0.224, 0.255] & 0.571 [0.503, 0.659] & 0.633 [0.505, 0.742] \\
11 +\bottomrule
12 +\end{tabular}
13
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere