# Research: LLM Structured Extraction (2025–2026 Best Practices) **Date of research:** 2026-08-05 **Method:** 4 targeted web searches (Claude structured outputs, OpenAI structured outputs, confidence scoring + human-in-the-loop, OCR+LLM invoice pipelines). **Purpose:** Design AIR's ingestion layer (`ingestion/`). **Core AIR principle restated:** the LLM only ever produces **AIR `EconomicEvent` documents** — never journal entries. Every LLM output is validated against the AIR JSON Schema, and low-confidence extractions are routed to a human approval queue before compilation. --- ## 1. Schema-constrained generation — Claude API The Claude Developer Platform supports **structured outputs** (public beta since late 2025, beta header `anthropic-beta: structured-outputs-2025-11-13`; initially Claude Sonnet 4.5 and Opus 4.1). Rather than merely prompting for JSON, the platform **compiles the JSON Schema into a grammar and constrains token generation during inference**, so responses are guaranteed to conform. Two modes: - **JSON outputs mode** — supply the schema via the `output_format` parameter; the response text is guaranteed schema-valid JSON. This is the fit for extraction tasks like "invoice PDF text → AIR Sale event". - **Strict tool use** — add `strict: true` to tool definitions; tool-call parameters exactly match the tool's input schema. This is the fit for the AIR **SDK-agent** path, where an agent calls `CreateEconomicEvent(...)` as a tool and the arguments are a schema-guaranteed AIR event. Established pre-structured-outputs pattern (still relevant as fallback for models/versions without the beta): define a single tool whose input schema is the extraction target and force the model to call it (tool use for extraction). Sources (consulted 2026-08-05): - Anthropic — Structured outputs on the Claude Developer Platform (blog): https://claude.com/blog/structured-outputs-on-the-claude-developer-platform - Claude Platform docs — Structured outputs: https://platform.claude.com/docs/en/build-with-claude/structured-outputs - Claude docs mirror — Structured outputs: https://docs.claude.com/en/docs/build-with-claude/structured-outputs ## 2. Schema-constrained generation — OpenAI OpenAI **Structured Outputs**: supply a JSON Schema via `response_format: {type: "json_schema", strict: true}` (or strict function calling). With `strict: true` the model **cannot** emit output violating the schema — required fields present, types correct, enum values valid. OpenAI reports ~100% schema compliance in evals vs ~86% for plain function calling and lower for raw JSON mode (JSON mode only guarantees *valid JSON*, not schema conformance). Supported on gpt-4o-2024-08-06 and later snapshots, gpt-4o-mini, o1 family and successors. **Takeaway for AIR:** both major providers now offer grammar-level schema enforcement. The AIR JSON Schema (source of truth in `schemas/`) can be passed directly as the constrained-decoding schema, making the ingestion layer **provider-agnostic**: one canonical schema → Claude `output_format` / strict tools, OpenAI `json_schema`, plus local re-validation. Sources (consulted 2026-08-05): - OpenAI — Introducing Structured Outputs in the API: https://openai.com/index/introducing-structured-outputs-in-the-api/ - OpenAI docs — Structured model outputs: https://developers.openai.com/api/docs/guides/structured-outputs ## 3. Validation: schema conformance is necessary, not sufficient Constrained decoding guarantees **shape**, not **truth**. 2025–2026 practice is to build validation in from the start (e.g. Pydantic models generated from the schema) and layer semantic checks after parse. For AIR, post-parse validation in the ingestion pipeline (before anything reaches the compiler): 1. **Schema validation** (defense in depth — never trust the provider's guarantee alone; also covers `air_version` compatibility). 2. **Semantic validators**: decimal amounts parse exactly (no floats), `qty × unit_price` consistent with line totals, gross vs net consistent, dates plausible, currency codes ISO 4217, jurisdiction codes known, referenced entities (`customer:...`, `company:...`) resolvable. 3. **Cross-document checks**: duplicate detection (hash + similarity vs previously ingested events) before the event enters the queue. 4. Only *then* the event enters Validate → (approval) → Compile. **The LLM's output is always an AIR event — a description of an economic event — never a journal entry; account selection, tax computation, and posting are the deterministic compiler's job.** Sources (consulted 2026-08-05): - Vellum — Document data extraction in 2026: LLMs vs OCRs: https://www.vellum.ai/blog/document-data-extraction-llms-vs-ocrs - Cleanlab — Real-time error detection for LLM structured outputs (benchmark): https://cleanlab.ai/blog/tlm-structured-outputs-benchmark/ ## 4. Confidence scoring and human-in-the-loop approval queues Current practice in document-AI products (Box Extract, LandingAI, IDP platforms): - **Field-level confidence scores**, not just document-level. Common techniques: self-consistency (sample multiple extractions and measure agreement — variance ⇒ low confidence), token log-probabilities where exposed, calibrated model-as-judge scoring (e.g. Cleanlab TLM), and OCR engine character/word confidences propagated to dependent fields. - **Threshold routing**: fields/documents **above** threshold flow straight through; **below** threshold they are routed to a **human review queue**. Reviewers verify or correct, and the queue size is proportional to actual uncertainty — that is what makes HITL tractable (review only the uncertain extractions, not everything). - **Pre-fill even when unsure**: show the extracted value to the reviewer — correcting a pre-filled value is faster than typing from scratch. - **Feedback loop**: corrections are logged and become evaluation/tuning data; thresholds are tuned per field criticality. ### AIR mapping AIR's schema already reserves `meta.llm: {model, confidence, reasoning_hash}` and `timestamps.approved` / `approver`. Concretely: - Store **per-field confidence** (e.g. `meta.llm.field_confidence: {total: 0.99, tax.codes: 0.71, ...}`) in addition to the overall score, plus the OCR score (`meta.source.ocr_score`). - **Approval policy is ALSL data**, not code: thresholds per event type, amount band, and field criticality (e.g. any event > X CAD, or `tax.jurisdiction` confidence < 0.9 ⇒ human approval). Below-threshold events sit in the approval queue with `timestamps.approved: null`; the compiler **refuses to compile unapproved events** whose policy requires approval. - Every approval/correction is an **audit-log entry** (hash-chained), and a human correction produces a new event version with provenance to the original extraction — the reviewer's identity lands in `approver`. Sources (consulted 2026-08-05): - Box — Confidence scores for Box Extract API: https://blog.box.com/confidence-scores-box-extract-api-know-when-rely-your-extractions - LandingAI — Building human-in-the-loop review workflows for document AI: https://landing.ai/llms/building-human-in-the-loop-review-workflows-for-document-ai - DEV — Human in the loop: using confidence scores for reliable document extraction: https://dev.to/iterationlayer/human-in-the-loop-using-confidence-scores-to-build-reliable-document-extraction-3pnb - Databricks — What is Human-in-the-Loop (HITL)?: https://www.databricks.com/blog/human-in-the-loop - Subhajit Bhar — Confidence scoring in document extraction: https://subhajitbhar.com/blog/idp/glossary/confidence-scoring-document-extraction/ ## 5. Invoice / document extraction pipelines (OCR + LLM) 2025–2026 consensus is a **hybrid architecture**: - **OCR / layout stage first** (or native PDF text extraction when the PDF has a text layer): specialized OCR + layout-analysis models (transformer-based document models) handle text recovery, tables, and segmentation, and yield **character/word confidence scores** that pure LLM vision calls don't expose reliably. - **LLM stage second**: semantic interpretation of the recovered text/layout into the target schema — the LLM is best at understanding ("3 chairs paid by Visa"), OCR/layout models at faithful transcription. For invoices specifically, hybrid splits are common (deterministic extraction for header fields, LLM for messy line items). - **Validation stage third**: schema + business-rule validation (see §3), then confidence-based routing (see §4). - Production systems combining specialized table extraction, layout analysis, and LLM semantic understanding get the best accuracy, at higher engineering cost — appropriate for AIR since extraction errors become financial records. ### AIR ingestion pipeline (Phase 4 target) ``` PDF/image/email → OCR + layout (per-field ocr confidence) [meta.source.ocr_score] → LLM extraction, schema-constrained to AIR [meta.llm.{model, confidence, field_confidence, reasoning_hash}] → schema + semantic validation (reject/repair) → duplicate detection → confidence routing: ≥ threshold → auto-approve per policy < threshold → human approval queue → approved AIR event → AIC compiler (deterministic) → journal entries ``` The LLM's role **ends** at the AIR event. No prompt, agent, or extraction step ever emits debits/credits; determinism, tax rules, and the double-entry invariant live entirely in the compiler. Sources (consulted 2026-08-05): - arXiv — Automated invoice data extraction using LLM and OCR (2511.05547): https://arxiv.org/abs/2511.05547 - Unstract — A 2026 guide to AI invoice data extraction: https://unstract.com/blog/ai-invoice-processing-and-data-extraction/ - Unstract — Invoice OCR in 2026: from document to accounting systems: https://unstract.com/blog/best-ocr-for-invoice-processing-invoice-ocr/ - AIMultiple — Invoice OCR benchmark: LLMs vs OCRs: https://aimultiple.com/invoice-ocr - Virtido — Document intelligence with LLMs (2026): https://virtido.com/blog/document-intelligence-llm-extraction-guide --- ## 6. Decisions / follow-ups - **D1:** Ingestion is provider-agnostic around one canonical AIR JSON Schema; use grammar-constrained structured outputs (Claude `output_format`/strict tools; OpenAI `json_schema` strict) with mandatory local re-validation. - **D2:** Per-field confidence + OCR score stored in `meta`; approval thresholds are versioned ALSL policies; compiler refuses unapproved events that policy flags. - **D3:** Hybrid OCR→LLM→validate→route pipeline; corrections feed an eval set for regression-testing extraction quality. - **Follow-up:** benchmark constrained vs unconstrained extraction accuracy on anonymized invoice fixtures (`tests/fixtures/`) before Phase 4; verify the current status of Anthropic's structured-outputs beta (header/model list may have changed since late 2025) at implementation time.