spb/wp45_uqo

Public

10commits
1branches
0releases
21.3 MBon disk
maindefault branch
22 min agolast push

progress: release note

Simon-Pierre Boucher committed 26 min ago (Nov 4, 2026) parent f48490e Browse files

2 changed files +5 −0

modified .gitignore +1 −0

				`@@ -45,3 +45,4 @@ experiments/outputs/raw/`
45	# evaluation fixtures contain copyrighted excerpts	45	# evaluation fixtures contain copyrighted excerpts
46	experiments/fixtures/heldout_v1/*/	46	experiments/fixtures/heldout_v1/*/
47	experiments/outputs/*/outputs/	47	experiments/outputs/*/outputs/
		48	+data/derived/index/shingles.json

modified docs/PROGRESS.md +4 −0

				`@@ -72,3 +72,7 @@ Verified milestones only. Times are America/Toronto unless marked UTC. Code vers`
72	## Phase 7 — Benchmark heldout_v1 run 2 (2026-11-03 20:35 → 21:40 ET) ✅ (supersedes run 1, archived)	72	## Phase 7 — Benchmark heldout_v1 run 2 (2026-11-03 20:35 → 21:40 ET) ✅ (supersedes run 1, archived)
73	- 199/199 runs ok (184 new + 10 reused OOD + 5 OOD referee re-run), 0 failures; 33 judge sets re-judged. Total API spend 93.2 USD (cap 110, A-005).	73	- 199/199 runs ok (184 new + 10 reused OOD + 5 OOD referee re-run), 0 failures; 33 judge sets re-judged. Total API spend 93.2 USD (cap 110, A-005).
74	- Outcomes (finance held-out, n = 24 paired): H1 D−B: clarity −0.12 [−0.31, 0.03], argument_logic +0.04 [−0.16, 0.22], contribution 0.00 [−0.28, 0.28] → not supported. H2 D−C: all \|Δ\| ≤ 0.12, intervals include 0 → not supported. H3 E−D: methodological_rigor +0.96 [0.44, 1.47] Holm p = 0.0085; fidelity +0.46 [0.03, 0.69] Holm p = 0.024 → supported; usefulness −0.33 [−0.78, 0.06] n.s. (trade-off). Exploratory E−A rigor +1.33 [0.84, 2.28] p = 0.002; E−B rigor +1.08 [0.56, 1.81] p = 0.009. Ablations: E − no_referee rigor +1.25 [0.63, 1.88], fidelity +0.56 [0.13, 1.13], argument_logic +0.50 [0.06, 0.94]; no_retrieval / no_profile / no_factcheck ≈ 0. H4: unverifiable citations ≈ 0 everywhere (−0.08 E−A); unsupported claims E−C −1.00 [−1.59, −0.16] (unadjusted p = 0.009), E−A −0.46, E−B −0.42 (n.s.); new numbers −0.17/−0.21/−0.42 (n.s.). H5: preregistered metric max 0.40 (self-overlap: improve_abstract 0.25, rewrite_literature 0.08); excluding the source document max 0.007, 100% < 0.02. Judge second pass (50 pairs): Spearman clarity 0.41, argument 0.58, contribution 0.84, rigor 0.80, fidelity 0.75, usefulness 0.53; MAD 0.26–0.40. Cost/latency: E 0.32 USD, 98 s vs D 0.15 USD, 49 s. OOD WP29 (n = 1): no E advantage.	74	- Outcomes (finance held-out, n = 24 paired): H1 D−B: clarity −0.12 [−0.31, 0.03], argument_logic +0.04 [−0.16, 0.22], contribution 0.00 [−0.28, 0.28] → not supported. H2 D−C: all \|Δ\| ≤ 0.12, intervals include 0 → not supported. H3 E−D: methodological_rigor +0.96 [0.44, 1.47] Holm p = 0.0085; fidelity +0.46 [0.03, 0.69] Holm p = 0.024 → supported; usefulness −0.33 [−0.78, 0.06] n.s. (trade-off). Exploratory E−A rigor +1.33 [0.84, 2.28] p = 0.002; E−B rigor +1.08 [0.56, 1.81] p = 0.009. Ablations: E − no_referee rigor +1.25 [0.63, 1.88], fidelity +0.56 [0.13, 1.13], argument_logic +0.50 [0.06, 0.94]; no_retrieval / no_profile / no_factcheck ≈ 0. H4: unverifiable citations ≈ 0 everywhere (−0.08 E−A); unsupported claims E−C −1.00 [−1.59, −0.16] (unadjusted p = 0.009), E−A −0.46, E−B −0.42 (n.s.); new numbers −0.17/−0.21/−0.42 (n.s.). H5: preregistered metric max 0.40 (self-overlap: improve_abstract 0.25, rewrite_literature 0.08); excluding the source document max 0.007, 100% < 0.02. Judge second pass (50 pairs): Spearman clarity 0.41, argument 0.58, contribution 0.84, rigor 0.80, fidelity 0.75, usefulness 0.53; MAD 0.26–0.40. Cost/latency: E 0.32 USD, 98 s vs D 0.15 USD, 49 s. OOD WP29 (n = 1): no E advantage.
		75	+
		76	+## Release (2026-11-03 21:55 ET)
		77	+- `paper/main.pdf` 75 pages, 0 LaTeX errors, 0 undefined citations/references; 29/29 tests; `README.md`, `docs/DELIVERY_REPORT.md` final; series `README.md` and `HARMONISATION.md` rows 45 added.
		78	+- Pushed to spbgit `gitsrv:srv/git/wp45_uqo.git` (history rewritten once to drop the 70 MB `shingles.json` before the public push; it is regenerated on demand).
75		79