spb/air Public MIT

1<!--2Project : AIR — Accounting Intermediate Representation3Author : Simon-Pierre Boucher4Contact : contact@spboucher.ai5File : llm-structured-extraction.md6-->78# Research: LLM Structured Extraction (2025–2026 Best Practices)910**Date of research:** 2026-08-0511**Method:** 4 targeted web searches (Claude structured outputs, OpenAI structured outputs, confidence scoring + human-in-the-loop, OCR+LLM invoice pipelines).12**Purpose:** Design AIR's ingestion layer (`ingestion/`). **Core AIR principle restated:** the LLM only ever produces **AIR `EconomicEvent` documents** — never journal entries. Every LLM output is validated against the AIR JSON Schema, and low-confidence extractions are routed to a human approval queue before compilation.1314---1516## 1. Schema-constrained generation — Claude API1718The Claude Developer Platform supports **structured outputs** (public beta since late 2025, beta header `anthropic-beta: structured-outputs-2025-11-13`; initially Claude Sonnet 4.5 and Opus 4.1). Rather than merely prompting for JSON, the platform **compiles the JSON Schema into a grammar and constrains token generation during inference**, so responses are guaranteed to conform.1920Two modes:21- **JSON outputs mode** — supply the schema via the `output_format` parameter; the response text is guaranteed schema-valid JSON. This is the fit for extraction tasks like "invoice PDF text → AIR Sale event".22- **Strict tool use** — add `strict: true` to tool definitions; tool-call parameters exactly match the tool's input schema. This is the fit for the AIR **SDK-agent** path, where an agent calls `CreateEconomicEvent(...)` as a tool and the arguments are a schema-guaranteed AIR event.2324Established pre-structured-outputs pattern (still relevant as fallback for models/versions without the beta): define a single tool whose input schema is the extraction target and force the model to call it (tool use for extraction).2526Sources (consulted 2026-08-05):27- Anthropic — Structured outputs on the Claude Developer Platform (blog): https://claude.com/blog/structured-outputs-on-the-claude-developer-platform28- Claude Platform docs — Structured outputs: https://platform.claude.com/docs/en/build-with-claude/structured-outputs29- Claude docs mirror — Structured outputs: https://docs.claude.com/en/docs/build-with-claude/structured-outputs3031## 2. Schema-constrained generation — OpenAI3233OpenAI **Structured Outputs**: supply a JSON Schema via `response_format: {type: "json_schema", strict: true}` (or strict function calling). With `strict: true` the model **cannot** emit output violating the schema — required fields present, types correct, enum values valid. OpenAI reports ~100% schema compliance in evals vs ~86% for plain function calling and lower for raw JSON mode (JSON mode only guarantees *valid JSON*, not schema conformance). Supported on gpt-4o-2024-08-06 and later snapshots, gpt-4o-mini, o1 family and successors.3435**Takeaway for AIR:** both major providers now offer grammar-level schema enforcement. The AIR JSON Schema (source of truth in `schemas/`) can be passed directly as the constrained-decoding schema, making the ingestion layer **provider-agnostic**: one canonical schema → Claude `output_format` / strict tools, OpenAI `json_schema`, plus local re-validation.3637Sources (consulted 2026-08-05):38- OpenAI — Introducing Structured Outputs in the API: https://openai.com/index/introducing-structured-outputs-in-the-api/39- OpenAI docs — Structured model outputs: https://developers.openai.com/api/docs/guides/structured-outputs4041## 3. Validation: schema conformance is necessary, not sufficient4243Constrained decoding guarantees **shape**, not **truth**. 2025–2026 practice is to build validation in from the start (e.g. Pydantic models generated from the schema) and layer semantic checks after parse. For AIR, post-parse validation in the ingestion pipeline (before anything reaches the compiler):44451. **Schema validation** (defense in depth — never trust the provider's guarantee alone; also covers `air_version` compatibility).462. **Semantic validators**: decimal amounts parse exactly (no floats), `qty × unit_price` consistent with line totals, gross vs net consistent, dates plausible, currency codes ISO 4217, jurisdiction codes known, referenced entities (`customer:...`, `company:...`) resolvable.473. **Cross-document checks**: duplicate detection (hash + similarity vs previously ingested events) before the event enters the queue.484. Only *then* the event enters Validate → (approval) → Compile. **The LLM's output is always an AIR event — a description of an economic event — never a journal entry; account selection, tax computation, and posting are the deterministic compiler's job.**4950Sources (consulted 2026-08-05):51- Vellum — Document data extraction in 2026: LLMs vs OCRs: https://www.vellum.ai/blog/document-data-extraction-llms-vs-ocrs52- Cleanlab — Real-time error detection for LLM structured outputs (benchmark): https://cleanlab.ai/blog/tlm-structured-outputs-benchmark/5354## 4. Confidence scoring and human-in-the-loop approval queues5556Current practice in document-AI products (Box Extract, LandingAI, IDP platforms):5758- **Field-level confidence scores**, not just document-level. Common techniques: self-consistency (sample multiple extractions and measure agreement — variance ⇒ low confidence), token log-probabilities where exposed, calibrated model-as-judge scoring (e.g. Cleanlab TLM), and OCR engine character/word confidences propagated to dependent fields.59- **Threshold routing**: fields/documents **above** threshold flow straight through; **below** threshold they are routed to a **human review queue**. Reviewers verify or correct, and the queue size is proportional to actual uncertainty — that is what makes HITL tractable (review only the uncertain extractions, not everything).60- **Pre-fill even when unsure**: show the extracted value to the reviewer — correcting a pre-filled value is faster than typing from scratch.61- **Feedback loop**: corrections are logged and become evaluation/tuning data; thresholds are tuned per field criticality.6263### AIR mapping6465AIR's schema already reserves `meta.llm: {model, confidence, reasoning_hash}` and `timestamps.approved` / `approver`. Concretely:6667- Store **per-field confidence** (e.g. `meta.llm.field_confidence: {total: 0.99, tax.codes: 0.71, ...}`) in addition to the overall score, plus the OCR score (`meta.source.ocr_score`).68- **Approval policy is ALSL data**, not code: thresholds per event type, amount band, and field criticality (e.g. any event > X CAD, or `tax.jurisdiction` confidence < 0.9 ⇒ human approval). Below-threshold events sit in the approval queue with `timestamps.approved: null`; the compiler **refuses to compile unapproved events** whose policy requires approval.69- Every approval/correction is an **audit-log entry** (hash-chained), and a human correction produces a new event version with provenance to the original extraction — the reviewer's identity lands in `approver`.7071Sources (consulted 2026-08-05):72- Box — Confidence scores for Box Extract API: https://blog.box.com/confidence-scores-box-extract-api-know-when-rely-your-extractions73- LandingAI — Building human-in-the-loop review workflows for document AI: https://landing.ai/llms/building-human-in-the-loop-review-workflows-for-document-ai74- DEV — Human in the loop: using confidence scores for reliable document extraction: https://dev.to/iterationlayer/human-in-the-loop-using-confidence-scores-to-build-reliable-document-extraction-3pnb75- Databricks — What is Human-in-the-Loop (HITL)?: https://www.databricks.com/blog/human-in-the-loop76- Subhajit Bhar — Confidence scoring in document extraction: https://subhajitbhar.com/blog/idp/glossary/confidence-scoring-document-extraction/7778## 5. Invoice / document extraction pipelines (OCR + LLM)79802025–2026 consensus is a **hybrid architecture**:8182- **OCR / layout stage first** (or native PDF text extraction when the PDF has a text layer): specialized OCR + layout-analysis models (transformer-based document models) handle text recovery, tables, and segmentation, and yield **character/word confidence scores** that pure LLM vision calls don't expose reliably.83- **LLM stage second**: semantic interpretation of the recovered text/layout into the target schema — the LLM is best at understanding ("3 chairs paid by Visa"), OCR/layout models at faithful transcription. For invoices specifically, hybrid splits are common (deterministic extraction for header fields, LLM for messy line items).84- **Validation stage third**: schema + business-rule validation (see §3), then confidence-based routing (see §4).85- Production systems combining specialized table extraction, layout analysis, and LLM semantic understanding get the best accuracy, at higher engineering cost — appropriate for AIR since extraction errors become financial records.8687### AIR ingestion pipeline (Phase 4 target)8889```90PDF/image/email91  → OCR + layout (per-field ocr confidence)        [meta.source.ocr_score]92  → LLM extraction, schema-constrained to AIR      [meta.llm.{model, confidence, field_confidence, reasoning_hash}]93  → schema + semantic validation (reject/repair)94  → duplicate detection95  → confidence routing:  ≥ threshold → auto-approve per policy96                         < threshold → human approval queue97  → approved AIR event → AIC compiler (deterministic) → journal entries98```99100The LLM's role **ends** at the AIR event. No prompt, agent, or extraction step ever emits debits/credits; determinism, tax rules, and the double-entry invariant live entirely in the compiler.101102Sources (consulted 2026-08-05):103- arXiv — Automated invoice data extraction using LLM and OCR (2511.05547): https://arxiv.org/abs/2511.05547104- Unstract — A 2026 guide to AI invoice data extraction: https://unstract.com/blog/ai-invoice-processing-and-data-extraction/105- Unstract — Invoice OCR in 2026: from document to accounting systems: https://unstract.com/blog/best-ocr-for-invoice-processing-invoice-ocr/106- AIMultiple — Invoice OCR benchmark: LLMs vs OCRs: https://aimultiple.com/invoice-ocr107- Virtido — Document intelligence with LLMs (2026): https://virtido.com/blog/document-intelligence-llm-extraction-guide108109---110111## 6. Decisions / follow-ups112113- **D1:** Ingestion is provider-agnostic around one canonical AIR JSON Schema; use grammar-constrained structured outputs (Claude `output_format`/strict tools; OpenAI `json_schema` strict) with mandatory local re-validation.114- **D2:** Per-field confidence + OCR score stored in `meta`; approval thresholds are versioned ALSL policies; compiler refuses unapproved events that policy flags.115- **D3:** Hybrid OCR→LLM→validate→route pipeline; corrections feed an eval set for regression-testing extraction quality.116- **Follow-up:** benchmark constrained vs unconstrained extraction accuracy on anonymized invoice fixtures (`tests/fixtures/`) before Phase 4; verify the current status of Anthropic's structured-outputs beta (header/model list may have changed since late 2025) at implementation time.117