SPB Git

Repository Public Research preview

DECISION_LOG.md 14.1 KB

# Decision log

Each entry records the decision, its rationale and the evidence. Entries D1–D14 were reconstructed on 2026-11-09 from the code and documents in the repository at commit 43899c8 plus the uncommitted work of that day.

ID Date Decision Rationale Evidence
D1 2026-11-09 The project lives in Recherche-UQO/ia/wp-ai-0002, with id AI-0002 in the "AI" series and origin gitsrv:srv/git/wpai0002_uqo.git follows AI-0001; no id collision CLAUDE.md
D2 2026-11-09 PMC is acquired only through the AWS Open Data bucket pmc-oa-opendata (per-article JSON + JATS, MD5-verified) PMC FTP packages and the OA Web Service were retired in August 2026; the bucket is the official bulk channel src/htr/ingest/pmc_s3.py, docs/gates/GATE_A.md
D3 2026-11-09 Pre-2018 status is verified per document from JATS <pub-date> (ppub → collection → epub …), the arXiv v1 date or book front matter; PMCID ranges are only a sampling prior; deposit and upload dates are never used mandate Phase 1: dates must be verified, not inferred parse/jats.py::_parse_pub_dates, ingest/arxiv_oai.py, ingest/books.py
D4 2026-11-09 Conservative licence policy: ND, NC-ND, TDM and ARXIV-NONEXCLUSIVE are tier C (no training by default); OTHER-OA and UNKNOWN are metadata only; the release set is CC0, PUBLIC-DOMAIN, CC-BY; the document's own licence statement overrides the provider field open access ≠ open licence; rewriting is a derivative work licenses/policy.py, ingest/corpus.py
D5 2026-11-09 Splits are made before rewriting. The unit is the near-duplicate cluster ∪ author group (venue optional); disciplines and periods can be held out whole; assignment is deterministic by hash prevents leakage through duplicates and authors data/splits.py, data/splits/smoke/splits_summary.json (group_leaks = [])
D6 2026-11-09 Placeholder protocol: citations, math and URLs are replaced by ⟦C/M/U n⟧ before any LLM call; rewrite and target are stored in placeholder form and restored at inference removes the most fragile spans from the rewriting task and makes preservation checkable parse/protect.py, rewrite/engine.py, inference/pipeline.py
D7 2026-11-09 Rewriters: claude-opus-5-5 is primary (effort low, Message Batches); claude-sonnet-5-5 and claude-haiku-5-5 are reserved as held-out models for RQ4; held-out prompts H1/H2 are never used in training; prompt version rw-1.0 pinned ids and prices; generalisation is a primary RQ configs/rewriting/pilot.json, rewrite/prompts.py
D8 2026-11-09 Label smoothing ε = 0.1 for L_token (0 in the overfit test) standard regulariser for seq2seq models/losses.py
D9 2026-11-09 Numbers are kept as text, not placeholders. They are checked strictly (no number missing, ≤ 1 extra) and up-weighted through L_factual numbers carry style (formatting, units) and must be learned; masking would hide factual errors validate/checks.py, training/data.py
D10 2026-11-09 From-scratch SentencePiece unigram tokenizer, trained on train-split text only, with atomic control and placeholder tokens avoids test leakage and third-party tokenizer licences; makes the tokenizer part of the release models/tokenizer.py, scripts/train_tokenizer.py
D11 2026-11-09 Pointer/copy gate over source positions, on by default (use_copy); --no-copy ablation most of the target is copyable from the rewrite; the mandate requires constrained preservation models/transformer.py, models/config.py
D12 2026-11-09 Pilot rewrite ceiling is 30 USD (hard stop), batch mode; anything larger needs explicit approval mandate: spend is opt-in configs/rewriting/pilot.json, docs/reproducibility/COMPUTE_BUDGET.md
D13 2026-11-09 The novelty claim is narrowed after the literature review: we do not claim to be first to invert LLM paraphrases (Rivera Soto et al. 2024, 2026); the claim is fidelity, rights-verified pre-2018 scientific originals, factual integrity, non-identifiability and memorisation controls, and discipline/era/section analysis verified literature docs/RESEARCH_GAP.md §5
D14 2026-11-09 Corpus scope widened: Medium stage (≥ 10 000 docs and ≥ 200 000 passages), economics and finance (arXiv q-fin/econ, Crossref CC BY journals), pre-2018 English reference books (Gutenberg, NCBI Bookshelf, OpenStax only if dated); a usable model release is a first-class deliverable; torch may be installed on other nodes author additions docs/MANDATE.md header, CLAUDE.md
D15 2026-11-09 Mock and dry-run providers are the default; real calls need --live; request ids are deterministic and results append-only resumability, no accidental spend rewrite/engine.py, scripts/rewrite.py
D16 2026-11-09 Model release format: SafeTensors + config.json + SentencePiece model + release metadata; usable API HTRReconstructor, deterministic decoding and preservation warnings mandate Phase 7 training/trainer.py::save_checkpoint, inference/pipeline.py

Pending decisions, to be logged when they are taken:

  • final SESOIs and the PREREGISTRATION v1.0 freeze hash;
  • the size labels vs the measured parameter counts (RESEARCH_PLAN §5);
  • the public release of weights trained on Claude rewrites (Anthropic terms, LICENSE_MATRIX.md §4);
  • the hosting namespace;
  • the B2 base model and its licence.

# D-FREEZE — 2026-11-09 17:59 UTC

PREREGISTRATION v1.0 frozen at commit e74b71266d5c0e8c20f181a50df139f52c771d0f (pre-specified in-session; SESOIs = v0.1 proposals, author confirmation pending). No confirmatory split had been scored with a model before this commit; only B0/B1 sanity checks on the smoke corpus.

# D17 — 2026-11-09 (after the exploratory validation of the M1 pilot)

Decision. Add a denoising pretraining stage on unlabelled, rights-cleared original passages (train split only; BART-style span masking, word dropout, local shuffling, rule-based paraphrase) before fine-tuning on (rewrite → original) pairs, and block repeated 4-grams at decoding. Why. The from-scratch M1-tiny scored chrF 32.9 vs 71.2 for identity on the validation split with degenerate repetitions: 3 900 pairs (≈ 1.7 M target tokens) cannot train a 27 M sequence model from random weights. Pretraining uses only data we already hold (no API spend) and is standard practice; it is reported as part of the model recipe, and token-loss-only fine-tuning without pretraining remains the documented control. Exploratory, before any confirmatory scoring.

# D18 — 2026-11-09 (pilot diagnostics)

Finding. Every M3 variant that included the factual term L_fact (weight 0.5, normalised by the number of protected/numeric tokens) failed to generalise (val CE ≈ 5.4 flat while train CE fell), and so did M1 + L_fact alone (pilot_m1_fact_tiny), whereas M3 token-only reached val CE 3.80 ≈ M1 (3.69). The culprit is therefore the normalisation of L_fact, not the dual-latent architecture nor the style/adversarial/contrastive terms. Decision. L_fact is now normalised by all target tokens (an extra weight λ_fact on protected tokens, bounded); presets unchanged. Seed variance on the pilot is large (M1 val CE 3.69 / 4.09 / 4.31 over seeds 1–3; M2 seed 2 = 5.03), so pilot-scale comparisons between architectures are not conclusive — the Medium stage is required.

# D19 — 2026-11-09 (single-term diagnostics)

Finding. L_contrastive alone (pilot_m3_cononly_tiny, val CE stuck ≈ 5.3) also prevented generalisation, L_semantic alone did not (best 4.08). Cause: six rewrites of the same original co-occur in length-bucketed batches and InfoNCE pushed their content vectors apart. Decision. Same-passage items are masked out of the InfoNCE negatives (passage_ids carried in batches). Presets unchanged; M3 and ablations are re-run as v3 after this fix.

D19 follow-up (v3 runs). With same-passage masking the M3 results are unchanged (full 5.31, no-style 4.82, no-adv 5.30, no-con 5.29, no-fact 5.30): only 1.3 % of off-diagonal pairs in length-bucketed batches share a passage, so the mask is correct but was not the cause. At pilot scale (≈ 3 900 pairs) every auxiliary term except L_semantic degrades generalisation, the style/adversarial branch most (val CE ≈ 5.3 vs 3.6–3.8 token-only). The dual-latent hypothesis therefore remains untested in a regime where it could work; it is carried to the Medium stage with the denoising-pretrained backbone and the loss weights re-tuned on validation (λ_style, λ_adv ≤ 0.05 as starting points), and reported as a negative pilot finding.

# D20 — 2026-11-10 (author decision)

Decision. The Medium-stage rewriter is claude-haiku-5-5 (author chose the Haiku option among Opus ≈ 1 070 / Sonnet ≈ 535 / Haiku ≈ 27 USD). Job medium_v1_haiku: 20 000 passages of medium_v1 (train 14 000 · val 2 000 · test 3 000 · held-out domain 500 · held-out period 500) × 6 generations = 120 000 requests, Message Batches, prompts rw-1.0, placeholder protocol 1.1. Consequence for the design. The training rewriter changes from Opus 5.5 (pilot) to Haiku 5.5; the Opus pilot pairs (974 passages, 5 383 validated) and the Sonnet/Haiku pilot held-out sets become the cross-model generalisation material (RQ4), and the paper reports rewriter strength as a factor. Validation thresholds (val-1.0) are unchanged; acceptance rates are reported per model.

# D21 — 2026-11-10 (author request)

Decision. Two additions: (i) RQ7 — detector sensitivity: originals, LLM rewrites and HTR reconstructions are scored by the commercial AI-text detector Pangram (API key in ~/Desktop/Cluster/secrets/pangram/, never in the repo; every call ledgered and cached). The measurement answers "how does a deployed detector react to inversion outputs"; a lower AI-likelihood is reported as detector behaviour, never as evidence of human authorship, and the dual-use implication (inversion as a paraphrase attack, cf. Krishna et al. 2023, Rivera Soto et al. 2024) is discussed in the ethics section. (ii) HTR-Edit, a non-generative, non-autoregressive text-editing model (LaserTagger/Felix family: per-token KEEP/DELETE tags + insertions from a phrase vocabulary learned from aligned rewrite→original pairs, encoder initialised from the Medium denoiser), which cannot hallucinate beyond its phrase vocabulary and leaves untouched spans byte-identical. Both are exploratory additions after the pre-registration freeze and are labelled as such.

# D22 — 2026-11-10 (author request: cloud GPU, "more instances", "a logic that scores higher")

Decision. (i) Full-scale training moves to rented NVIDIA H100 SXM instances (Lambda Cloud, us-southeast-1; author provided the account and key; billing 4.29 USD/h per 1×H100 and 8.38 USD/h per 2×H100; scripts/lambda_*.sh): instance 1 (1×H100) HTR-Base (335 M) denoising pretraining 30 000 steps → pair fine-tuning → predictions; instance 2 (2×H100) HTR-Large (1.07 B, 40 000 + 30 000 steps, 8 192 / 6 144-token batches) and HTR-Base seed 2; instance 3 (2×H100) the "HTR 2.0" tracks below. The cluster (MPS) keeps the Tiny/Small/seed/HTR-Edit runs. (ii) HTR 2.0 — four pre-declared, exploratory tracks to raise the lexical score above the identity ceiling, each reported against the same B0/B1/M* numbers, all tuned on validation only and applied once to test: T1 decoding (nucleus sampling + MBR with chrF++ utility + source-anchored reranking; scripts/decode_search.py); T2 minimum-risk training (Shen et al. 2016: expected 1 − chrF++ + fact penalty over {gold, greedy, 8 samples}, CE anchor; htr.training.mrt, scripts/train_mrt.py); T3 pretrained backbone (STRAP-style inverse paraphrase, Krishna et al. 2020: google/flan-t5-large 780 M fine-tuned on the same pairs/splits, system B2_flan-t5-large; pre-2018 contamination of the backbone's pretraining data is a validity threat to be stated); T4 discriminator (FacebookAI/roberta-large original-vs-rewrite on the train split, scripts/train_discriminator.py) as a reranking signal and as a reward for a KL-regularised policy-optimisation variant (HUMPA, ICLR 2025 / AuthorMist 2025 recipes with a source-fidelity term). The confirmatory results (Gate E) are not re-run; HTR 2.0 systems enter the paper as exploratory additions with their own validation-tuned choices disclosed. Why. The frozen M* beats identity by only +0.44 chrF++ (below SESOI); the author asked for a mechanism that scores materially higher. The literature review (STRAP; HUMPA; AuthorMist; Adversarial Paraphrasing, UMD 2025; MBR/MRT) points to pretrained priors and direct metric optimisation, not to architecture. Ceiling stated up front: the best-of-9 oracle over Tiny samples is only +1.7 over identity on validation, while the perfect-edit oracle (HTR-Edit labels) is +13: the headroom is in knowing the edit, not in picking among samples. A "100 %" reconstruction is impossible by construction (non-identifiability, RESEARCH_PLAN §2).

# D23 — 2026-11-11 (author decision: public release of the weights)

Decision. The author asked to publish the six served models on the Hugging Face Hub under spboucher04/ (htr-tiny, htr-base, flan-t5-large-htr, flan-t5-xl-htr, qwen2.5-1.5b-htr, qwen2.5-3b-htr; scripts/export_hf_cards.py, scripts/hf_publish.py; token in ~/Desktop/Cluster/secrets/huggingface/.env, created through the author's logged-in Safari session). Model cards carry the synthetic-output notice, the no-detection/no-integrity-use clause, the training-data provenance (rights-verified pre-2018 originals; Claude Haiku 5.5 rewrites; pairs not released), the pretraining-contamination caveat for the backbones, the Pangram block with its caveat, and the metrics of docs/STATUS.md. Licence chosen for the fine-tuned weights: CC BY-NC 4.0 (non-commercial research use), because the training pairs include CC BY-NC sources and LLM outputs; qwen2.5-3b-htr additionally inherits the Qwen Research Licence; the htr inference code shipped inside htr-tiny/htr-base stays Apache-2.0. The open question of LICENSE_MATRIX.md §4 (provider terms on training with Claude outputs) is thereby decided by the author for the weights; the pairs themselves remain unreleased.

Files main
Menu

Theme