Medium rewrite plan config and calibrated cost estimate; STATUS medium_v1

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Simon-Pierre Boucher committed 2 h ago (Nov 9, 2026) parent 0bfd2d4 Browse files

3 changed files +38 −5

added configs/rewriting/medium.json +15 −0

++{
++  "name": "medium-v1",
++  "provider": "anthropic",
++  "model": "claude-opus-5-5",
++  "heldout_models": ["claude-sonnet-5-5", "claude-haiku-5-5"],
++  "variants": ["A", "B", "C", "D", "E"],
++  "samples_per_variant": {"A": 1, "B": 1, "C": 1, "D": 1, "E": 2},
++  "max_tokens": 1500,
++  "effort": "low",
++  "mode": "batch",
++  "concurrency": 4,
++  "budget_usd_ceiling": 600.0,
++  "allocation": {"train": 14000, "val": 2000, "test": 3000, "heldout_domain": 500, "heldout_period": 500},
++  "notes": "Medium stage (Gate E): 20 000 passages x 6 generations on corpus medium_v1, stratified by discipline x section role within each split. REQUIRES EXPLICIT AUTHOR APPROVAL BEFORE --live. Ceiling is a hard stop. Alternative: claude-sonnet-5-5 at about half the cost."
++}

modified docs/STATUS.md +15 −2

				`@@ -19,6 +19,16 @@`
19	\| Total \| \| 6 744 \| \| 29.68 \|	19	\| Total \| \| 6 744 \| \| 29.68 \|
20	Measured mean tokens per request on Opus: 691 in / 323 out → 0.0046 USD per request in batch mode.	20	Measured mean tokens per request on Opus: 691 in / 323 out → 0.0046 USD per request in batch mode.
21		21
		22	+## Medium corpus v1 (final acquisition, built 2026-11-09 night; `data/parsed/medium_v1/build_summary.json`, splits `data/splits/medium_v1/` with v0 assignments preserved)
		23	+\| Item \| Value \|
		24	+\|---\|---\|
		25	+\| Documents kept \| 14 470 (14 900 stubs; 2 parse errors, 35 post-2017, 61 undated, 36 licence-excluded, 16 non-English, 234 exact + 46 near duplicates) \|
		26	+\| By source \| PMC 10 911 · Crossref econ/finance 1 730 · Bookshelf 1 286 chapters · Gutenberg 448 · arXiv 90 · OpenStax 5 \|
		27	+\| By licence \| CC-BY 9 702 · CC-BY-NC 1 849 · PUBLIC-DOMAIN 1 737 · CC-BY-NC-SA 1 070 · CC0 109 · CC-BY-SA 3 \|
		28	+\| Disciplines \| biomedicine 8 828 · life sciences 2 222 · economics 1 289 · finance 564 · social sciences 363 · engineering 252 · chemistry 174 · CS 168 · earth 147 · physics 102 · statistics 84 · general reference 58 · humanities 56 · mathematics 45 · other 118 \|
		29	+\| Paragraphs / eligible passages \| 897 419 / 451 072 \|
		30	+\| Splits \| train 11 035 · val 1 270 · test 1 261 · heldout_domain 253 · heldout_period 510 · quarantine 141 (fixed-split conflicts) · 0 leaks \|
		31	+
22	## Medium corpus v0 (built 2026-11-09 evening; `data/parsed/medium_v0/build_summary.json`, splits `data/splits/medium_v0/`)	32	## Medium corpus v0 (built 2026-11-09 evening; `data/parsed/medium_v0/build_summary.json`, splits `data/splits/medium_v0/`)
23	\| Item \| Value \|	33	\| Item \| Value \|
24	\|---\|---\|	34	\|---\|---\|
				`@@ -47,8 +57,11 @@ Acquisition still running when v0 was built: Crossref (895/1 730), Gutenberg (28`
47	Crossref text-mining links for CC BY content. Confirm this policy (documented in `docs/RISK_REGISTER.md`).	57	Crossref text-mining links for CC BY content. Confirm this policy (documented in `docs/RISK_REGISTER.md`).
48	2. Anthropic terms: publishing weights trained on Claude rewrites — recorded as a risk to confirm, not a conclusion.	58	2. Anthropic terms: publishing weights trained on Claude rewrites — recorded as a risk to confirm, not a conclusion.
49	3. SESOI thresholds in `docs/PREREGISTRATION.md` v0.1 must be confirmed before the v1.0 freeze.	59	3. SESOI thresholds in `docs/PREREGISTRATION.md` v0.1 must be confirmed before the v1.0 freeze.
50	−4. Budget for the Medium-stage rewrites: 20 000 passages × 6 generations ≈ 550 USD at measured Opus token usage	60	+4. Budget for the Medium-stage rewrites (Gate E) — approval required before `--live`. Plan `configs/rewriting/medium.json`:
51	− (batch); ≈ 275 USD with Sonnet 5.5; needs explicit approval.	61	+ 20 000 passages of `medium_v1` (train 14 000 · val 2 000 · test 3 000 · held-out domain 500 · held-out period 500) × 6
		62	+ generations = 120 000 requests. Pilot-calibrated estimate (Medium passages are longer than the PMC pilot's):
		63	+ ≈ 1 070 USD with Opus 5.5 (batch), ≈ 535 USD with Sonnet 5.5, ≈ 27 USD with Haiku 5.5; a 10 000-passage plan halves these.
		64	+ Per-request lower bound from the pilot's actual spend: 551 / 276 / 14 USD. Nothing has been submitted.
52		65
53	## Known limitations at this point	66	## Known limitations at this point
54	- Discipline labels for PMC are heuristic (journal title / subject categories); relabelled once (165/1 180 changed).	67	- Discipline labels for PMC are heuristic (journal title / subject categories); relabelled once (165/1 180 changed).

modified src/htr/rewrite/engine.py +8 −3

				`@@ -96,11 +96,16 @@ class RewriteEngine:`
96	self.ledger[f"n_{res.status}"] = self.ledger.get(f"n_{res.status}", 0) + 1	96	self.ledger[f"n_{res.status}"] = self.ledger.get(f"n_{res.status}", 0) + 1
97	return rw	97	return rw
98		98
		99	+ # Calibration from the pilot job (5 801 Opus 5.5 requests, 2026-11-09): measured 691 input / 323 output tokens per
		100	+ # request vs the naive chars/4 estimate (492 / 259) → multiply by 1.40 / 1.25.
		101	+ EST_IN_FACTOR = 1.40
		102	+ EST_OUT_FACTOR = 1.25
		103	+
99	def estimate(self, reqs: list[RewriteRequest], passages: dict[str, dict], pricing: dict \| None) -> dict:	104	def estimate(self, reqs: list[RewriteRequest], passages: dict[str, dict], pricing: dict \| None) -> dict:
100	− """Rough pre-flight estimate (chars/4 ≈ tokens; output ≈ 1.1× input passage; prompt overhead ≈ 260 tokens)."""	105	+ """Pre-flight estimate: chars/4 ≈ tokens, prompt overhead ≈ 260 tokens, calibrated on the pilot (see constants)."""
101	pend = [r for r in reqs if r.request_id not in self.done]	106	pend = [r for r in reqs if r.request_id not in self.done]
102	− toks_in = sum(260 + len(_protected_text(passages[r.passage_id])) // 4 for r in pend)	107	+ toks_in = int(self.EST_IN_FACTOR * sum(260 + len(_protected_text(passages[r.passage_id])) // 4 for r in pend))
103	− toks_out = sum(int(1.1 * len(passages[r.passage_id]["text"]) // 4) for r in pend)	108	+ toks_out = int(self.EST_OUT_FACTOR * sum(int(1.1 * len(passages[r.passage_id]["text"]) // 4) for r in pend))
104	est = None	109	est = None
105	if pricing and self.provider.model in pricing["models"]:	110	if pricing and self.provider.model in pricing["models"]:
106	m = pricing["models"][self.provider.model]	111	m = pricing["models"][self.provider.model]
107		112

spb/wpai0002_uqo

Medium rewrite plan config and calibrated cost estimate; STATUS medium_v1

Theme