SPB Git

spb/air Public MIT

AIR — The Language of Accounting.

languageaccountingcompiler
Python 100%
10.9 KB
View source Raw Blame History

# Research: LLM Structured Extraction (2025–2026 Best Practices)

Date of research: 2026-08-05 Method: 4 targeted web searches (Claude structured outputs, OpenAI structured outputs, confidence scoring + human-in-the-loop, OCR+LLM invoice pipelines). Purpose: Design AIR's ingestion layer (ingestion/). Core AIR principle restated: the LLM only ever produces AIR EconomicEvent documents — never journal entries. Every LLM output is validated against the AIR JSON Schema, and low-confidence extractions are routed to a human approval queue before compilation.

# 1. Schema-constrained generation — Claude API

The Claude Developer Platform supports structured outputs (public beta since late 2025, beta header anthropic-beta: structured-outputs-2025-11-13; initially Claude Sonnet 4.5 and Opus 4.1). Rather than merely prompting for JSON, the platform compiles the JSON Schema into a grammar and constrains token generation during inference, so responses are guaranteed to conform.

Two modes:

  • JSON outputs mode — supply the schema via the output_format parameter; the response text is guaranteed schema-valid JSON. This is the fit for extraction tasks like "invoice PDF text → AIR Sale event".
  • Strict tool use — add strict: true to tool definitions; tool-call parameters exactly match the tool's input schema. This is the fit for the AIR SDK-agent path, where an agent calls CreateEconomicEvent(...) as a tool and the arguments are a schema-guaranteed AIR event.

Established pre-structured-outputs pattern (still relevant as fallback for models/versions without the beta): define a single tool whose input schema is the extraction target and force the model to call it (tool use for extraction).

Sources (consulted 2026-08-05):

# 2. Schema-constrained generation — OpenAI

OpenAI Structured Outputs: supply a JSON Schema via response_format: {type: "json_schema", strict: true} (or strict function calling). With strict: true the model cannot emit output violating the schema — required fields present, types correct, enum values valid. OpenAI reports ~100% schema compliance in evals vs ~86% for plain function calling and lower for raw JSON mode (JSON mode only guarantees valid JSON, not schema conformance). Supported on gpt-4o-2024-08-06 and later snapshots, gpt-4o-mini, o1 family and successors.

Takeaway for AIR: both major providers now offer grammar-level schema enforcement. The AIR JSON Schema (source of truth in schemas/) can be passed directly as the constrained-decoding schema, making the ingestion layer provider-agnostic: one canonical schema → Claude output_format / strict tools, OpenAI json_schema, plus local re-validation.

Sources (consulted 2026-08-05):

# 3. Validation: schema conformance is necessary, not sufficient

Constrained decoding guarantees shape, not truth. 2025–2026 practice is to build validation in from the start (e.g. Pydantic models generated from the schema) and layer semantic checks after parse. For AIR, post-parse validation in the ingestion pipeline (before anything reaches the compiler):

  1. Schema validation (defense in depth — never trust the provider's guarantee alone; also covers air_version compatibility).
  2. Semantic validators: decimal amounts parse exactly (no floats), qty × unit_price consistent with line totals, gross vs net consistent, dates plausible, currency codes ISO 4217, jurisdiction codes known, referenced entities (customer:..., company:...) resolvable.
  3. Cross-document checks: duplicate detection (hash + similarity vs previously ingested events) before the event enters the queue.
  4. Only then the event enters Validate → (approval) → Compile. The LLM's output is always an AIR event — a description of an economic event — never a journal entry; account selection, tax computation, and posting are the deterministic compiler's job.

Sources (consulted 2026-08-05):

# 4. Confidence scoring and human-in-the-loop approval queues

Current practice in document-AI products (Box Extract, LandingAI, IDP platforms):

  • Field-level confidence scores, not just document-level. Common techniques: self-consistency (sample multiple extractions and measure agreement — variance ⇒ low confidence), token log-probabilities where exposed, calibrated model-as-judge scoring (e.g. Cleanlab TLM), and OCR engine character/word confidences propagated to dependent fields.
  • Threshold routing: fields/documents above threshold flow straight through; below threshold they are routed to a human review queue. Reviewers verify or correct, and the queue size is proportional to actual uncertainty — that is what makes HITL tractable (review only the uncertain extractions, not everything).
  • Pre-fill even when unsure: show the extracted value to the reviewer — correcting a pre-filled value is faster than typing from scratch.
  • Feedback loop: corrections are logged and become evaluation/tuning data; thresholds are tuned per field criticality.

# AIR mapping

AIR's schema already reserves meta.llm: {model, confidence, reasoning_hash} and timestamps.approved / approver. Concretely:

  • Store per-field confidence (e.g. meta.llm.field_confidence: {total: 0.99, tax.codes: 0.71, ...}) in addition to the overall score, plus the OCR score (meta.source.ocr_score).
  • Approval policy is ALSL data, not code: thresholds per event type, amount band, and field criticality (e.g. any event > X CAD, or tax.jurisdiction confidence < 0.9 ⇒ human approval). Below-threshold events sit in the approval queue with timestamps.approved: null; the compiler refuses to compile unapproved events whose policy requires approval.
  • Every approval/correction is an audit-log entry (hash-chained), and a human correction produces a new event version with provenance to the original extraction — the reviewer's identity lands in approver.

Sources (consulted 2026-08-05):

# 5. Invoice / document extraction pipelines (OCR + LLM)

2025–2026 consensus is a hybrid architecture:

  • OCR / layout stage first (or native PDF text extraction when the PDF has a text layer): specialized OCR + layout-analysis models (transformer-based document models) handle text recovery, tables, and segmentation, and yield character/word confidence scores that pure LLM vision calls don't expose reliably.
  • LLM stage second: semantic interpretation of the recovered text/layout into the target schema — the LLM is best at understanding ("3 chairs paid by Visa"), OCR/layout models at faithful transcription. For invoices specifically, hybrid splits are common (deterministic extraction for header fields, LLM for messy line items).
  • Validation stage third: schema + business-rule validation (see §3), then confidence-based routing (see §4).
  • Production systems combining specialized table extraction, layout analysis, and LLM semantic understanding get the best accuracy, at higher engineering cost — appropriate for AIR since extraction errors become financial records.

# AIR ingestion pipeline (Phase 4 target)

text
PDF/image/email
  → OCR + layout (per-field ocr confidence)        [meta.source.ocr_score]
  → LLM extraction, schema-constrained to AIR      [meta.llm.{model, confidence, field_confidence, reasoning_hash}]
  → schema + semantic validation (reject/repair)
  → duplicate detection
  → confidence routing:  ≥ threshold → auto-approve per policy
                         < threshold → human approval queue
  → approved AIR event → AIC compiler (deterministic) → journal entries

The LLM's role ends at the AIR event. No prompt, agent, or extraction step ever emits debits/credits; determinism, tax rules, and the double-entry invariant live entirely in the compiler.

Sources (consulted 2026-08-05):

# 6. Decisions / follow-ups

  • D1: Ingestion is provider-agnostic around one canonical AIR JSON Schema; use grammar-constrained structured outputs (Claude output_format/strict tools; OpenAI json_schema strict) with mandatory local re-validation.
  • D2: Per-field confidence + OCR score stored in meta; approval thresholds are versioned ALSL policies; compiler refuses unapproved events that policy flags.
  • D3: Hybrid OCR→LLM→validate→route pipeline; corrections feed an eval set for regression-testing extraction quality.
  • Follow-up: benchmark constrained vs unconstrained extraction accuracy on anonymized invoice fixtures (tests/fixtures/) before Phase 4; verify the current status of Anthropic's structured-outputs beta (header/model list may have changed since late 2025) at implementation time.