spb/wpai0002_uqo

1# Pangram detector-sensitivity evaluation23**What this is.** A *measurement of how one commercial AI-text detector (Pangram Labs) behaves* on three kinds of text:4the pre-2018 human originals (`target_text`), the LLM rewrites (`source_text`) and the reconstructions produced by our5systems. **What it is not.** A detector score is not evidence of authorship, and a lower AI score is not evidence that6a text is "human" (project rule 3, `docs/MANDATE.md`). A system that lowers a detector's score is, at the same time, a7potential detector-evasion tool; this is reported as a dual-use finding (see § Dual use). Pangram scores are never used8as a training signal, loss, model-selection criterion or reward.910Code: `src/htr/evaluation/pangram.py` (client), `scripts/pangram_eval.py` (evaluation), tests11`tests/unit/test_pangram_client.py` (offline, HTTP mocked). Outputs: `artifacts/metrics/<name>/pangram_report.json`,12`pangram_summary.md`, `pangram_items.jsonl` (per item scores + text sha256, no text).1314## 1. Verified API contract (retrieved 2026-11-10)1516| Fact | Value | Source |17|---|---|---|18| Base URL (text) | `https://text.external-api.pangram.com` (other pages/SDK mention `text.api.pangram.com` / `text.api.pangramlabs.com`; the API reference uses `text.external-api…`, which we confirmed live) | <https://docs.pangram.com/api-reference/introduction> |19| Auth | header `x-api-key: <key>`; 401 missing/invalid key | same |20| Single text | `POST /task {text, model, public_dashboard_link?}` → `{task_id}`; poll `GET /task/{task_id}` until `stage ∈ {STAGE_SUCCESS, STAGE_FAILED}` (asynchronous; no synchronous endpoint is documented) | <https://docs.pangram.com/api-reference/ai-detection> |21| Model selector | `GET /models` → our key: `["default", "pangram-4", "pangram-4.1"]`. We pin `pangram-4.1` explicitly (the docs advise sending `model`; minor versions may change behind a selector; `pangram-4` is currently served by 4.1 per the API's own notice) | live call + ai-detection page |22| Document fields | `prediction_short` ∈ {`AI`, `Human`, `Mixed`} (Pangram 4); `headline` (observed: `Human Written`, `AI Assisted`, `AI Detected`); `prediction` (sentence); `fraction_ai`, `fraction_ai_assisted`, `fraction_human` ∈ [0,1]; `num_*_segments`; `version` | ai-detection page |23| Window fields | `label` ∈ {`AI-Generated`, `AI-Assisted`, `Human Written`}; `ai_assistance_score` ∈ [0,1] (0 fully human … 1 fully AI-generated); `confidence` ∈ {High, Medium, Low}; `start_index`, `end_index`, `word_count`, `token_length`, `is_humanized`, `humanizer_score`; 4.1 adds `class_probabilities` [Human, AI-Assisted, AI-Generated] (live response also carries an undocumented `calibrated_confidence`) | ai-detection page + live call |24| Bulk | `POST /bulk {items:[{id,text}], model}` → 202 `{bulk_id, accepted_items, failed_items}`; `GET /bulk/{id}` status `queued/running/succeeded/failed/partial`; `GET /bulk/{id}/results?offset&limit≤1000`; results kept 48 h; **launch limit 1 000 billable units per request** (413 above) | <https://docs.pangram.com/api-reference/bulk-api.md> |25| Billing unit | one *started* 100-word block per text (Pangram 4), ≥ 1 unit per text | bulk-api page |26| Price | text (Pangram 4.1) **0.05 USD per 100 words**; **bulk text API 20 % discount** (→ 0.04 USD/unit); consumer plans include 10 USD (Individual) / 200 USD (Professional) of API credit per month | <https://www.pangram.com/pricing> |27| Rate limits | not published; 429 when the key's configured limit is exceeded; 402 insufficient credits | introduction page |28| Text length | no minimum/maximum documented; 422 for invalid text | introduction page |29| Terms of use | Terms of Service (last updated 2025-08-14): licence "solely for your internal business use"; prohibits reproducing/creating derivative works of the Service, circumventing security/access controls, reselling access, illegal use. **No clause on benchmarking, comparative testing, publishing results, research use, or on using outputs to train models** was found. Our use (scoring our own research texts through the official API, aggregate reporting) does not fall under any prohibition we could find; we do not redistribute raw Pangram outputs. Publication of aggregate results should be re-checked with the owner before the paper | <https://www.pangram.com/terms-of-service> |3031**Live schema check (2026-11-10 17:07 UTC).** One 37-word, 2-sentence text through `POST /task` (`model:32"pangram-4"`): 200, notice "pangram-4 … is now served by pangram-4.1", result `STAGE_SUCCESS` in 2.6 s, `version33"4.1"`, `prediction_short "Human"`, one window with all documented fields. Logged in the ledger (`run: schema_probe`).3435### Field semantics used here3637* `ai_likelihood` (our field) = word-weighted mean of the windows' `ai_assistance_score` (fallback `fraction_ai`).38  It is a continuous detector score, not a calibrated probability of AI authorship.39* `% AI` = share with the API's own document label `prediction_short == "AI"`.40* `% not Human` = share labelled `AI` **or** `Mixed`. Observed mapping: a text judged "entirely AI-assisted" gets41  `prediction_short = "Mixed"` (headline `AI Assisted`), so `% AI` is a very strict criterion for short passages;42  `% not Human` is the operative "flagged by Pangram" rate in this study.4344## 2. Implementation4546* Key: `$PANGRAM_API_KEY` or `~/Desktop/Cluster/secrets/pangram/.env`; never printed, never written under the repo.47* Cache: `data/manifests/pangram_cache/<sha[:2]>/<sha256(model, text)>.json` (raw response). The client writes a48  `.gitignore` (`*`) in that directory because raw responses echo the scored (licensed) text. A text is never billed twice.49* Bulk jobs: the id→text map is written to `pangram_cache/pending_bulk/<bulk_id>.json` *before* polling; a crashed or50  timed-out run is resumed on the next invocation without re-submission.51* Ledger: `data/manifests/pangram_ledger.jsonl` (gitignored): timestamp, endpoint, model, n texts, chars, words,52  estimated units and USD, status, run name, task/bulk id. No text, no key. Pangram does not return the billed amount,53  so costs are list-price estimates.54* Retries with exponential backoff (honours `Retry-After`) on 429/5xx and transport errors; minimum interval between55  requests; `max_calls` guard (texts submitted per client); the script also aborts *before any call* if the56  cache-aware cost estimate exceeds `--max-usd` (default 20) and runs only with `--live`.57* Placeholder restoration: `protect(passage.text, version=metadata.protect_version)` is re-run on the source passage;58  if it reproduces `target_text` exactly, its pools restore ⟦C/M/U k⟧ in every condition via59  `restore_placeholders`; unresolved placeholders become a neutral marker `[k]` (counted in the report). In both runs60  below all passages were found, `protect()` reproduced 100 % of targets, and **0** placeholders needed a neutral marker.61* Sampling: one pair per passage, proportional stratification on `metadata.discipline` (largest remainder),62  deterministic hash order, seed 20261110. CIs: `htr.evaluation.bootstrap` (2 000 resamples, clustered by `doc_id`);63  paired differences via `paired_bootstrap_diff`. When all paired differences are exactly 0 the bootstrap p-value of64  that helper is degenerate and is reported as n/a.6566## 3. Runs (2026-11-10)6768Budget rule: the owner capped each run at ~20 USD. At 0.04 USD per started 100-word block (bulk) and ~1.8 units per69text, the requested sizes would have cost ~72.7 USD (Medium test, n = 200) and ~22.7 USD (Opus, n = 100), so the70samples were reduced to **n = 54** and **n = 85** (dry-run estimates 19.40 and 19.48 USD). B0_identity outputs are71byte-identical to the rewrites, so they are served from the cache (scored once, billed once).7273Total billed: 1 + 269 + 254 = **524 texts**, ~973 units, **~38.93 USD** (schema probe 0.05 + 19.40 + 19.48), all in74the ledger. No bulk item failed; no task error.7576### 3.1 `pangram_test` — Medium test split, Haiku rewrites (n = 54 passages, 50 documents, 13 disciplines)7778`scripts/pangram_eval.py --pairs data/pairs/medium_v1_haiku/validated/pairs.jsonl --outputs-dir artifacts/predictions/medium_test --systems B0_identity,htr-m1-tiny-s3,htr-m1-small,B2_t5-small --n 54 --name pangram_test --live`7980| Condition | mean ai_likelihood [95 % CI] | max | % not Human (AI or Mixed) [95 % CI] | % AI | labels |81|---|---|---:|---|---:|---|82| original (pre-2018) | 0.0000 [0.0000, 0.0000] | 0.0000 | 0.0 [0.0, 0.0] | 0 | Human 54 |83| rewrite (Haiku) = B0_identity | 0.1875 [0.1245, 0.2532] | 0.84 | 46.3 [32.1, 60.0] | 0 | Human 29, Mixed 25 |84| htr-m1-tiny-s3 | 0.0000 [0.0000, 0.0000] | 0.0001 | 0.0 [0.0, 0.0] | 0 | Human 54 |85| htr-m1-small | 0.0000 [0.0000, 0.0000] | 0.0000 | 0.0 [0.0, 0.0] | 0 | Human 54 |86| B2_t5-small | 0.0318 [0.0039, 0.0668] | 0.70 | 7.4 [1.9, 14.5] | 0 | Human 50, Mixed 4 |8788Paired (system − rewrite) Δ ai_likelihood: htr-m1-tiny-s3 −0.1875 [−0.2532, −0.1245]; htr-m1-small −0.187589[−0.2532, −0.1245]; B2_t5-small −0.1556 [−0.2202, −0.0929]. Label transitions rewrite → system: both HTR models90Mixed→Human 25/25; B2 Mixed→Human 22, Mixed→Mixed 3, Human→Mixed 1. System − original: HTR Δ ≈ 0 (all Human);91B2 +0.0318 [0.0039, 0.0668].9293### 3.2 `pangram_opus` — held-out rewriting model (Claude Opus pilot rewrites; n = 85 passages, 40 documents)9495`scripts/pangram_eval.py --pairs data/pairs/pilot_opus/validated/pairs.jsonl --outputs-dir artifacts/predictions/heldout_opus --systems B0_identity,htr-m1-tiny-s3 --n 85 --name pangram_opus --passages data/parsed/pilot_pmc/passages.jsonl,data/parsed/medium_v1/passages.jsonl --live`9697| Condition | mean ai_likelihood [95 % CI] | max | % not Human [95 % CI] | % AI | labels |98|---|---|---:|---|---:|---|99| original | 0.0000 [0.0000, 0.0000] | 0.0000 | 0.0 [0.0, 0.0] | 0 | Human 85 |100| rewrite (Opus) = B0_identity | 0.1210 [0.0783, 0.1769] | 0.50 | 32.9 [22.5, 45.5] | 0 | Human 57, Mixed 28 |101| htr-m1-tiny-s3 | 0.0000 [0.0000, 0.0000] | 0.0000 | 0.0 [0.0, 0.0] | 0 | Human 85 |102103Paired htr-m1-tiny-s3 − rewrite: −0.1210 [−0.1769, −0.0783]; Mixed→Human 28/28.104105## 4. Reading and dual use106107* Pangram 4.1 labelled **every** pre-2018 original "Human" (139/139) and flagged **33–46 %** of the LLM rewrites as108  `Mixed` (mostly "AI-assisted"); **no** passage of any condition received the document label `AI`, so the strict109  `% AI` metric is uninformative at this passage length.110* After HTR reconstruction, Pangram's flags disappeared in all sampled cases (25/25 and 28/28 Mixed→Human), although111  the HTR outputs stay close to the rewrites (mean chrF to the rewrite ≈ 84–85 on these samples; chrF to the112  original 70–72, i.e. ≈ identity, consistent with `artifacts/metrics/conf_test`). T5-small (B2) removed most but not113  all flags.114* This shows that **light, learned edits suffice to move this detector's output**; it says nothing about whether the115  reconstructions are "human" — the same outputs remain near-copies of the LLM rewrites, and verbatim/stylometric116  fidelity to the originals is measured elsewhere (chrF, W1, probes). The result is a **dual-use risk**: a117  reconstruction model can act as a detector-evasion "humaniser". It must be stated as such in the paper and model118  card (see `docs/MANDATE.md`, honest claims), the model must not be optimised against Pangram or any detector, and119  any release decision should weigh it. We did not test Pangram's dedicated `is_humanized` head beyond recording it120  (0 humanised windows observed in all 523 scored texts).121122## 5. Caveats123124* Small samples (54 and 85 passages, one detector, one model version 4.1); CIs are wide for proportions.125* Passages are short (mean ~130–150 words, mostly one Pangram window); detectors are generally less decisive on126  short texts, and Pangram publishes no minimum length.127* Restoration re-inserts LaTeX math (`$…$`) and citation strings exactly as in the source passage, identically for all128  conditions, so it cannot drive between-condition differences.129* Pangram is a moving target (selectors are remapped server-side); results are tied to `version` recorded per item.130* Costs are list-price estimates from the ledger, not invoices.131