spb/wpai0002_uqo

1# Pangram detector-sensitivity evaluation — `pangram_test`23Model `pangram-4.1` (served version(s) 4.1); pairs `data/pairs/medium_v1_haiku/validated/pairs.jsonl`; outputs `artifacts/predictions/medium_test`; n = 54 pairs (50 documents), stratified by discipline, seed 20261110.45**This measures one commercial detector's behaviour. A low AI-likelihood is not evidence of human authorship.**67`ai_likelihood` = word-weighted mean of Pangram window `ai_assistance_score` (0 = human … 1 = AI-generated). `% AI` = share labelled `AI` by the API (`prediction_short`); `% not Human` = `AI` or `Mixed`. 95 % CIs: document-clustered bootstrap.89| Condition | n | mean ai_likelihood [95 % CI] | median | % score > 0 | % AI [95 % CI] | % not Human [95 % CI] | labels |10|---|---:|---|---:|---:|---|---|---|11| original | 54 | 0.0000 [0.0000, 0.0000] | 0.0000 | 27.8 | 0.0 [0.0, 0.0] | 0.0 [0.0, 0.0] | Human 54 |12| rewrite | 54 | 0.1875 [0.1245, 0.2532] | 0.0000 | 96.3 | 0.0 [0.0, 0.0] | 46.3 [32.1, 60.0] | Human 29, Mixed 25 |13| B0_identity | 54 | 0.1875 [0.1245, 0.2532] | 0.0000 | 96.3 | 0.0 [0.0, 0.0] | 46.3 [32.1, 60.0] | Human 29, Mixed 25 |14| htr-m1-tiny-s3 | 54 | 0.0000 [0.0000, 0.0000] | 0.0000 | 51.9 | 0.0 [0.0, 0.0] | 0.0 [0.0, 0.0] | Human 54 |15| htr-m1-small | 54 | 0.0000 [0.0000, 0.0000] | 0.0000 | 48.1 | 0.0 [0.0, 0.0] | 0.0 [0.0, 0.0] | Human 54 |16| B2_t5-small | 54 | 0.0318 [0.0039, 0.0668] | 0.0000 | 68.5 | 0.0 [0.0, 0.0] | 7.4 [1.9, 14.5] | Human 50, Mixed 4 |1718## Paired differences (system − reference)1920| Contrast | n | Δ ai_likelihood [95 % CI] | p | Δ % AI [95 % CI] | p | label transitions (ref→system) |21|---|---:|---|---:|---|---:|---|22| B0_identity - rewrite | 54 | 0.0000 [0.0000, 0.0000] | n/a (all Δ = 0) | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 29, Mixed->Mixed 25 |23| B0_identity - original | 54 | 0.1875 [0.1245, 0.2532] | 0.000 | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 29, Human->Mixed 25 |24| htr-m1-tiny-s3 - rewrite | 54 | -0.1875 [-0.2532, -0.1245] | 0.000 | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 29, Mixed->Human 25 |25| htr-m1-tiny-s3 - original | 54 | 0.0000 [0.0000, 0.0000] | 0.001 | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 54 |26| htr-m1-small - rewrite | 54 | -0.1875 [-0.2532, -0.1245] | 0.000 | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 29, Mixed->Human 25 |27| htr-m1-small - original | 54 | 0.0000 [0.0000, 0.0000] | 0.018 | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 54 |28| B2_t5-small - rewrite | 54 | -0.1556 [-0.2202, -0.0929] | 0.000 | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 28, Human->Mixed 1, Mixed->Human 22, Mixed->Mixed 3 |29| B2_t5-small - original | 54 | 0.0318 [0.0039, 0.0668] | 0.000 | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 50, Human->Mixed 4 |3031## Placeholder restoration3233Passages found 54/54; protect() re-run reproduces target_text: 54/54; placeholders restored 504, replaced by a neutral marker 0 (by kind {}); texts scored with neutral markers: 0.3435## Cost3637Whole run (ledger, all invocations): 269 texts billed in 1 request(s), ~485 units, ~19.40 USD. This invocation: 0 submitted, 269 cache hits. Pangram does not return the billed amount per call; estimate = started 100-word blocks × list price (bulk 0.04 USD/unit).38