Pangram detector-sensitivity evaluation — pangram_opus
Model pangram-4.1 (served version(s) 4.1); pairs data/pairs/pilot_opus/validated/pairs.jsonl; outputs artifacts/predictions/heldout_opus; n = 85 pairs (40 documents), stratified by discipline, seed 20261110.
This measures one commercial detector's behaviour. A low AI-likelihood is not evidence of human authorship.
ai_likelihood = word-weighted mean of Pangram window ai_assistance_score (0 = human … 1 = AI-generated). % AI = share labelled AI by the API (prediction_short); % not Human = AI or Mixed. 95 % CIs: document-clustered bootstrap.
| Condition | n | mean ai_likelihood [95 % CI] | median | % score > 0 | % AI [95 % CI] | % not Human [95 % CI] | labels |
|---|---|---|---|---|---|---|---|
| original | 85 | 0.0000 [0.0000, 0.0000] | 0.0000 | 18.8 | 0.0 [0.0, 0.0] | 0.0 [0.0, 0.0] | Human 85 |
| rewrite | 85 | 0.1210 [0.0783, 0.1769] | 0.0000 | 91.8 | 0.0 [0.0, 0.0] | 32.9 [22.5, 45.5] | Human 57, Mixed 28 |
| B0_identity | 85 | 0.1210 [0.0783, 0.1769] | 0.0000 | 91.8 | 0.0 [0.0, 0.0] | 32.9 [22.5, 45.5] | Human 57, Mixed 28 |
| htr-m1-tiny-s3 | 85 | 0.0000 [0.0000, 0.0000] | 0.0000 | 36.5 | 0.0 [0.0, 0.0] | 0.0 [0.0, 0.0] | Human 85 |
Paired differences (system − reference)
| Contrast | n | Δ ai_likelihood [95 % CI] | p | Δ % AI [95 % CI] | p | label transitions (ref→system) |
|---|---|---|---|---|---|---|
| B0_identity - rewrite | 85 | 0.0000 [0.0000, 0.0000] | n/a (all Δ = 0) | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 57, Mixed->Mixed 28 |
| B0_identity - original | 85 | 0.1210 [0.0783, 0.1769] | 0.000 | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 57, Human->Mixed 28 |
| htr-m1-tiny-s3 - rewrite | 85 | -0.1210 [-0.1769, -0.0783] | 0.000 | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 57, Mixed->Human 28 |
| htr-m1-tiny-s3 - original | 85 | 0.0000 [0.0000, 0.0000] | 0.001 | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 85 |
Placeholder restoration
Passages found 85/85; protect() re-run reproduces target_text: 85/85; placeholders restored 575, replaced by a neutral marker 0 (by kind {}); texts scored with neutral markers: 0.
Cost
Whole run (ledger, all invocations): 254 texts billed in 1 request(s), ~487 units, ~19.48 USD. This invocation: 254 submitted, 0 cache hits. Pangram does not return the billed amount per call; estimate = started 100-word blocks × list price (bulk 0.04 USD/unit).