Pangram detector-sensitivity evaluation — pangram_test
Model pangram-4.1 (served version(s) 4.1); pairs data/pairs/medium_v1_haiku/validated/pairs.jsonl; outputs artifacts/predictions/medium_test; n = 54 pairs (50 documents), stratified by discipline, seed 20261110.
This measures one commercial detector's behaviour. A low AI-likelihood is not evidence of human authorship.
ai_likelihood = word-weighted mean of Pangram window ai_assistance_score (0 = human … 1 = AI-generated). % AI = share labelled AI by the API (prediction_short); % not Human = AI or Mixed. 95 % CIs: document-clustered bootstrap.
| Condition | n | mean ai_likelihood [95 % CI] | median | % score > 0 | % AI [95 % CI] | % not Human [95 % CI] | labels |
|---|---|---|---|---|---|---|---|
| original | 54 | 0.0000 [0.0000, 0.0000] | 0.0000 | 27.8 | 0.0 [0.0, 0.0] | 0.0 [0.0, 0.0] | Human 54 |
| rewrite | 54 | 0.1875 [0.1245, 0.2532] | 0.0000 | 96.3 | 0.0 [0.0, 0.0] | 46.3 [32.1, 60.0] | Human 29, Mixed 25 |
| B0_identity | 54 | 0.1875 [0.1245, 0.2532] | 0.0000 | 96.3 | 0.0 [0.0, 0.0] | 46.3 [32.1, 60.0] | Human 29, Mixed 25 |
| htr-m1-tiny-s3 | 54 | 0.0000 [0.0000, 0.0000] | 0.0000 | 51.9 | 0.0 [0.0, 0.0] | 0.0 [0.0, 0.0] | Human 54 |
| htr-m1-small | 54 | 0.0000 [0.0000, 0.0000] | 0.0000 | 48.1 | 0.0 [0.0, 0.0] | 0.0 [0.0, 0.0] | Human 54 |
| B2_t5-small | 54 | 0.0318 [0.0039, 0.0668] | 0.0000 | 68.5 | 0.0 [0.0, 0.0] | 7.4 [1.9, 14.5] | Human 50, Mixed 4 |
Paired differences (system − reference)
| Contrast | n | Δ ai_likelihood [95 % CI] | p | Δ % AI [95 % CI] | p | label transitions (ref→system) |
|---|---|---|---|---|---|---|
| B0_identity - rewrite | 54 | 0.0000 [0.0000, 0.0000] | n/a (all Δ = 0) | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 29, Mixed->Mixed 25 |
| B0_identity - original | 54 | 0.1875 [0.1245, 0.2532] | 0.000 | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 29, Human->Mixed 25 |
| htr-m1-tiny-s3 - rewrite | 54 | -0.1875 [-0.2532, -0.1245] | 0.000 | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 29, Mixed->Human 25 |
| htr-m1-tiny-s3 - original | 54 | 0.0000 [0.0000, 0.0000] | 0.001 | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 54 |
| htr-m1-small - rewrite | 54 | -0.1875 [-0.2532, -0.1245] | 0.000 | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 29, Mixed->Human 25 |
| htr-m1-small - original | 54 | 0.0000 [0.0000, 0.0000] | 0.018 | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 54 |
| B2_t5-small - rewrite | 54 | -0.1556 [-0.2202, -0.0929] | 0.000 | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 28, Human->Mixed 1, Mixed->Human 22, Mixed->Mixed 3 |
| B2_t5-small - original | 54 | 0.0318 [0.0039, 0.0668] | 0.000 | 0.0 [0.0, 0.0] | n/a (all Δ = 0) | Human->Human 50, Human->Mixed 4 |
Placeholder restoration
Passages found 54/54; protect() re-run reproduces target_text: 54/54; placeholders restored 504, replaced by a neutral marker 0 (by kind {}); texts scored with neutral markers: 0.
Cost
Whole run (ledger, all invocations): 269 texts billed in 1 request(s), ~485 units, ~19.40 USD. This invocation: 0 submitted, 269 cache hits. Pangram does not return the billed amount per call; estimate = started 100-word blocks × list price (bulk 0.04 USD/unit).