spb/wpai0002_uqo

1# Pangram detector-sensitivity evaluation — `pangram_opus`23Model `pangram-4.1` (served version(s) 4.1); pairs `data/pairs/pilot_opus/validated/pairs.jsonl`; outputs `artifacts/predictions/heldout_opus`; n = 85 pairs (40 documents), stratified by discipline, seed 20261110.45**This measures one commercial detector's behaviour. A low AI-likelihood is not evidence of human authorship.**67`ai_likelihood` = word-weighted mean of Pangram window `ai_assistance_score` (0 = human … 1 = AI-generated). `% AI` = share labelled `AI` by the API (`prediction_short`); `% not Human` = `AI` or `Mixed`. 95 % CIs: document-clustered bootstrap.89| Condition | n | mean ai_likelihood [95 % CI] | median | % score > 0 | % AI [95 % CI] | % not Human [95 % CI] | labels |10|---|---:|---|---:|---:|---|---|---|11| original | 85 | 0.0000 [0.0000, 0.0000] | 0.0000 | 18.8 | 0.0 [0.0, 0.0] | 0.0 [0.0, 0.0] | Human 85 |12| rewrite | 85 | 0.1210 [0.0783, 0.1769] | 0.0000 | 91.8 | 0.0 [0.0, 0.0] | 32.9 [22.5, 45.5] | Human 57, Mixed 28 |13| B0_identity | 85 | 0.1210 [0.0783, 0.1769] | 0.0000 | 91.8 | 0.0 [0.0, 0.0] | 32.9 [22.5, 45.5] | Human 57, Mixed 28 |14| htr-m1-tiny-s3 | 85 | 0.0000 [0.0000, 0.0000] | 0.0000 | 36.5 | 0.0 [0.0, 0.0] | 0.0 [0.0, 0.0] | Human 85 |1516## Paired differences (system − reference)1718| Contrast | n | Δ ai_likelihood [95 % CI] | p | Δ % AI [95 % CI] | p | label transitions (ref→system) |19|---|---:|---|---:|---|---:|---|20| B0_identity - rewrite | 85 | 0.0000 [0.0000, 0.0000] | n/a (all Δ = 0) | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 57, Mixed->Mixed 28 |21| B0_identity - original | 85 | 0.1210 [0.0783, 0.1769] | 0.000 | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 57, Human->Mixed 28 |22| htr-m1-tiny-s3 - rewrite | 85 | -0.1210 [-0.1769, -0.0783] | 0.000 | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 57, Mixed->Human 28 |23| htr-m1-tiny-s3 - original | 85 | 0.0000 [0.0000, 0.0000] | 0.001 | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 85 |2425## Placeholder restoration2627Passages found 85/85; protect() re-run reproduces target_text: 85/85; placeholders restored 575, replaced by a neutral marker 0 (by kind {}); texts scored with neutral markers: 0.2829## Cost3031Whole run (ledger, all invocations): 254 texts billed in 1 request(s), ~487 units, ~19.48 USD. This invocation: 254 submitted, 0 cache hits. Pangram does not return the billed amount per call; estimate = started 100-word blocks × list price (bulk 0.04 USD/unit).32