Medium rewrite plan config and calibrated cost estimate; STATUS medium_v1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
3 changed files +38 −5
added
configs/rewriting/medium.json
+15 −0
@@ -0,0 +1,15 @@ | ||
| 1 | +{ | |
| 2 | + "name": "medium-v1", | |
| 3 | + "provider": "anthropic", | |
| 4 | + "model": "claude-opus-5-5", | |
| 5 | + "heldout_models": ["claude-sonnet-5-5", "claude-haiku-5-5"], | |
| 6 | + "variants": ["A", "B", "C", "D", "E"], | |
| 7 | + "samples_per_variant": {"A": 1, "B": 1, "C": 1, "D": 1, "E": 2}, | |
| 8 | + "max_tokens": 1500, | |
| 9 | + "effort": "low", | |
| 10 | + "mode": "batch", | |
| 11 | + "concurrency": 4, | |
| 12 | + "budget_usd_ceiling": 600.0, | |
| 13 | + "allocation": {"train": 14000, "val": 2000, "test": 3000, "heldout_domain": 500, "heldout_period": 500}, | |
| 14 | + "notes": "Medium stage (Gate E): 20 000 passages x 6 generations on corpus medium_v1, stratified by discipline x section role within each split. REQUIRES EXPLICIT AUTHOR APPROVAL BEFORE --live. Ceiling is a hard stop. Alternative: claude-sonnet-5-5 at about half the cost." | |
| 15 | +} | |
modified
docs/STATUS.md
+15 −2
@@ -19,6 +19,16 @@ | ||
| 19 | 19 | | **Total** | | **6 744** | | **29.68** | |
| 20 | 20 | Measured mean tokens per request on Opus: 691 in / 323 out → 0.0046 USD per request in batch mode. |
| 21 | 21 | |
| 22 | +## Medium corpus v1 (final acquisition, built 2026-11-09 night; `data/parsed/medium_v1/build_summary.json`, splits `data/splits/medium_v1/` with v0 assignments preserved) | |
| 23 | +| Item | Value | | |
| 24 | +|---|---| | |
| 25 | +| Documents kept | **14 470** (14 900 stubs; 2 parse errors, 35 post-2017, 61 undated, 36 licence-excluded, 16 non-English, 234 exact + 46 near duplicates) | | |
| 26 | +| By source | PMC 10 911 · Crossref econ/finance **1 730** · Bookshelf 1 286 chapters · Gutenberg 448 · arXiv 90 · OpenStax 5 | | |
| 27 | +| By licence | CC-BY 9 702 · CC-BY-NC 1 849 · PUBLIC-DOMAIN 1 737 · CC-BY-NC-SA 1 070 · CC0 109 · CC-BY-SA 3 | | |
| 28 | +| Disciplines | biomedicine 8 828 · life sciences 2 222 · **economics 1 289 · finance 564** · social sciences 363 · engineering 252 · chemistry 174 · CS 168 · earth 147 · physics 102 · statistics 84 · general reference 58 · humanities 56 · mathematics 45 · other 118 | | |
| 29 | +| Paragraphs / **eligible passages** | 897 419 / **451 072** | | |
| 30 | +| Splits | train 11 035 · val 1 270 · test 1 261 · heldout_domain 253 · heldout_period 510 · quarantine 141 (fixed-split conflicts) · 0 leaks | | |
| 31 | + | |
| 22 | 32 | ## Medium corpus v0 (built 2026-11-09 evening; `data/parsed/medium_v0/build_summary.json`, splits `data/splits/medium_v0/`) |
| 23 | 33 | | Item | Value | |
| 24 | 34 | |---|---| |
@@ -47,8 +57,11 @@ Acquisition still running when v0 was built: Crossref (895/1 730), Gutenberg (28 | ||
| 47 | 57 | Crossref text-mining links for CC BY content. Confirm this policy (documented in `docs/RISK_REGISTER.md`). |
| 48 | 58 | 2. Anthropic terms: publishing weights trained on Claude rewrites — recorded as a risk to confirm, not a conclusion. |
| 49 | 59 | 3. SESOI thresholds in `docs/PREREGISTRATION.md` v0.1 must be confirmed before the v1.0 freeze. |
| 50 | −4. Budget for the Medium-stage rewrites: 20 000 passages × 6 generations ≈ 550 USD at measured Opus token usage | |
| 51 | − (batch); ≈ 275 USD with Sonnet 5.5; needs explicit approval. | |
| 60 | +4. **Budget for the Medium-stage rewrites (Gate E) — approval required before `--live`.** Plan `configs/rewriting/medium.json`: | |
| 61 | + 20 000 passages of `medium_v1` (train 14 000 · val 2 000 · test 3 000 · held-out domain 500 · held-out period 500) × 6 | |
| 62 | + generations = 120 000 requests. Pilot-calibrated estimate (Medium passages are longer than the PMC pilot's): | |
| 63 | + **≈ 1 070 USD with Opus 5.5 (batch)**, ≈ 535 USD with Sonnet 5.5, ≈ 27 USD with Haiku 5.5; a 10 000-passage plan halves these. | |
| 64 | + Per-request lower bound from the pilot's actual spend: 551 / 276 / 14 USD. Nothing has been submitted. | |
| 52 | 65 | |
| 53 | 66 | ## Known limitations at this point |
| 54 | 67 | - Discipline labels for PMC are heuristic (journal title / subject categories); relabelled once (165/1 180 changed). |
modified
src/htr/rewrite/engine.py
+8 −3
@@ -96,11 +96,16 @@ class RewriteEngine: | ||
| 96 | 96 | self.ledger[f"n_{res.status}"] = self.ledger.get(f"n_{res.status}", 0) + 1 |
| 97 | 97 | return rw |
| 98 | 98 | |
| 99 | + # Calibration from the pilot job (5 801 Opus 5.5 requests, 2026-11-09): measured 691 input / 323 output tokens per | |
| 100 | + # request vs the naive chars/4 estimate (492 / 259) → multiply by 1.40 / 1.25. | |
| 101 | + EST_IN_FACTOR = 1.40 | |
| 102 | + EST_OUT_FACTOR = 1.25 | |
| 103 | + | |
| 99 | 104 | def estimate(self, reqs: list[RewriteRequest], passages: dict[str, dict], pricing: dict | None) -> dict: |
| 100 | − """Rough pre-flight estimate (chars/4 ≈ tokens; output ≈ 1.1× input passage; prompt overhead ≈ 260 tokens).""" | |
| 105 | + """Pre-flight estimate: chars/4 ≈ tokens, prompt overhead ≈ 260 tokens, calibrated on the pilot (see constants).""" | |
| 101 | 106 | pend = [r for r in reqs if r.request_id not in self.done] |
| 102 | − toks_in = sum(260 + len(_protected_text(passages[r.passage_id])) // 4 for r in pend) | |
| 103 | − toks_out = sum(int(1.1 * len(passages[r.passage_id]["text"]) // 4) for r in pend) | |
| 107 | + toks_in = int(self.EST_IN_FACTOR * sum(260 + len(_protected_text(passages[r.passage_id])) // 4 for r in pend)) | |
| 108 | + toks_out = int(self.EST_OUT_FACTOR * sum(int(1.1 * len(passages[r.passage_id]["text"]) // 4) for r in pend)) | |
| 104 | 109 | est = None |
| 105 | 110 | if pricing and self.provider.model in pricing["models"]: |
| 106 | 111 | m = pricing["models"][self.provider.model] |
| 107 | 112 | |