# Pangram detector-sensitivity evaluation

What this is. A measurement of how one commercial AI-text detector (Pangram Labs) behaves on three kinds of text: the pre-2018 human originals (target_text), the LLM rewrites (source_text) and the reconstructions produced by our systems. What it is not. A detector score is not evidence of authorship, and a lower AI score is not evidence that a text is "human" (project rule 3, docs/MANDATE.md). A system that lowers a detector's score is, at the same time, a potential detector-evasion tool; this is reported as a dual-use finding (see § Dual use). Pangram scores are never used as a training signal, loss, model-selection criterion or reward.

Code: src/htr/evaluation/pangram.py (client), scripts/pangram_eval.py (evaluation), tests tests/unit/test_pangram_client.py (offline, HTTP mocked). Outputs: artifacts/metrics/<name>/pangram_report.json, pangram_summary.md, pangram_items.jsonl (per item scores + text sha256, no text).

# 1. Verified API contract (retrieved 2026-11-10)

Live schema check (2026-11-10 17:07 UTC). One 37-word, 2-sentence text through POST /task (model: "pangram-4"): 200, notice "pangram-4 … is now served by pangram-4.1", result STAGE_SUCCESS in 2.6 s, version "4.1", prediction_short "Human", one window with all documented fields. Logged in the ledger (run: schema_probe).

# Field semantics used here

• ai_likelihood (our field) = word-weighted mean of the windows' ai_assistance_score (fallback fraction_ai). It is a continuous detector score, not a calibrated probability of AI authorship.
• % AI = share with the API's own document label prediction_short == "AI".
• % not Human = share labelled AI or Mixed. Observed mapping: a text judged "entirely AI-assisted" gets prediction_short = "Mixed" (headline AI Assisted), so % AI is a very strict criterion for short passages; % not Human is the operative "flagged by Pangram" rate in this study.

# 2. Implementation

• Key: $PANGRAM_API_KEY or ~/Desktop/Cluster/secrets/pangram/.env; never printed, never written under the repo.
• Cache: data/manifests/pangram_cache/<sha[:2]>/<sha256(model, text)>.json (raw response). The client writes a .gitignore (*) in that directory because raw responses echo the scored (licensed) text. A text is never billed twice.
• Bulk jobs: the id→text map is written to pangram_cache/pending_bulk/<bulk_id>.json before polling; a crashed or timed-out run is resumed on the next invocation without re-submission.
• Ledger: data/manifests/pangram_ledger.jsonl (gitignored): timestamp, endpoint, model, n texts, chars, words, estimated units and USD, status, run name, task/bulk id. No text, no key. Pangram does not return the billed amount, so costs are list-price estimates.
• Retries with exponential backoff (honours Retry-After) on 429/5xx and transport errors; minimum interval between requests; max_calls guard (texts submitted per client); the script also aborts before any call if the cache-aware cost estimate exceeds --max-usd (default 20) and runs only with --live.
• Placeholder restoration: protect(passage.text, version=metadata.protect_version) is re-run on the source passage; if it reproduces target_text exactly, its pools restore ⟦C/M/U k⟧ in every condition via restore_placeholders; unresolved placeholders become a neutral marker [k] (counted in the report). In both runs below all passages were found, protect() reproduced 100 % of targets, and 0 placeholders needed a neutral marker.
• Sampling: one pair per passage, proportional stratification on metadata.discipline (largest remainder), deterministic hash order, seed 20261110. CIs: htr.evaluation.bootstrap (2 000 resamples, clustered by doc_id); paired differences via paired_bootstrap_diff. When all paired differences are exactly 0 the bootstrap p-value of that helper is degenerate and is reported as n/a.

# 3. Runs (2026-11-10)

Budget rule: the owner capped each run at ~20 USD. At 0.04 USD per started 100-word block (bulk) and ~1.8 units per text, the requested sizes would have cost ~72.7 USD (Medium test, n = 200) and ~22.7 USD (Opus, n = 100), so the samples were reduced to n = 54 and n = 85 (dry-run estimates 19.40 and 19.48 USD). B0_identity outputs are byte-identical to the rewrites, so they are served from the cache (scored once, billed once).

Total billed: 1 + 269 + 254 = 524 texts, ~973 units, ~38.93 USD (schema probe 0.05 + 19.40 + 19.48), all in the ledger. No bulk item failed; no task error.

# 3.1 pangram_test — Medium test split, Haiku rewrites (n = 54 passages, 50 documents, 13 disciplines)

scripts/pangram_eval.py --pairs data/pairs/medium_v1_haiku/validated/pairs.jsonl --outputs-dir artifacts/predictions/medium_test --systems B0_identity,htr-m1-tiny-s3,htr-m1-small,B2_t5-small --n 54 --name pangram_test --live

Paired (system − rewrite) Δ ai_likelihood: htr-m1-tiny-s3 −0.1875 [−0.2532, −0.1245]; htr-m1-small −0.1875 [−0.2532, −0.1245]; B2_t5-small −0.1556 [−0.2202, −0.0929]. Label transitions rewrite → system: both HTR models Mixed→Human 25/25; B2 Mixed→Human 22, Mixed→Mixed 3, Human→Mixed 1. System − original: HTR Δ ≈ 0 (all Human); B2 +0.0318 [0.0039, 0.0668].

# 3.2 pangram_opus — held-out rewriting model (Claude Opus pilot rewrites; n = 85 passages, 40 documents)

scripts/pangram_eval.py --pairs data/pairs/pilot_opus/validated/pairs.jsonl --outputs-dir artifacts/predictions/heldout_opus --systems B0_identity,htr-m1-tiny-s3 --n 85 --name pangram_opus --passages data/parsed/pilot_pmc/passages.jsonl,data/parsed/medium_v1/passages.jsonl --live

Paired htr-m1-tiny-s3 − rewrite: −0.1210 [−0.1769, −0.0783]; Mixed→Human 28/28.

# 4. Reading and dual use

• Pangram 4.1 labelled every pre-2018 original "Human" (139/139) and flagged 33–46 % of the LLM rewrites as Mixed (mostly "AI-assisted"); no passage of any condition received the document label AI, so the strict % AI metric is uninformative at this passage length.
• After HTR reconstruction, Pangram's flags disappeared in all sampled cases (25/25 and 28/28 Mixed→Human), although the HTR outputs stay close to the rewrites (mean chrF to the rewrite ≈ 84–85 on these samples; chrF to the original 70–72, i.e. ≈ identity, consistent with artifacts/metrics/conf_test). T5-small (B2) removed most but not all flags.
• This shows that light, learned edits suffice to move this detector's output; it says nothing about whether the reconstructions are "human" — the same outputs remain near-copies of the LLM rewrites, and verbatim/stylometric fidelity to the originals is measured elsewhere (chrF, W1, probes). The result is a dual-use risk: a reconstruction model can act as a detector-evasion "humaniser". It must be stated as such in the paper and model card (see docs/MANDATE.md, honest claims), the model must not be optimised against Pangram or any detector, and any release decision should weigh it. We did not test Pangram's dedicated is_humanized head beyond recording it (0 humanised windows observed in all 523 scored texts).

# 5. Caveats

• Small samples (54 and 85 passages, one detector, one model version 4.1); CIs are wide for proportions.
• Passages are short (mean ~130–150 words, mostly one Pangram window); detectors are generally less decisive on short texts, and Pangram publishes no minimum length.
• Restoration re-inserts LaTeX math ($…$) and citation strings exactly as in the source passage, identically for all conditions, so it cannot drive between-condition differences.
• Pangram is a moving target (selectors are remapped server-side); results are tied to version recorded per item.
• Costs are list-price estimates from the ledger, not invoices.