SPB Git

Repository Public

intelligence.md 22.1 KB

# On-device assistance (ORIX Intelligence)

ORIX Intelligence is an optional panel that uses the language model built into macOS to explain and suggest. It never computes. Every piece of model output is either verified by deterministic code (the formula parser, the calculation engine, the auditor, the data profile) or is displayed as text that the user may copy. When the system model is not available the panel says why, and the rest of the application is unaffected.

Package Role Imports
Packages/SheetIntelligence Intelligence actor, model backend, context builder, prompts, validators and post-checks SheetCore; the Audit and Statistics products of SheetAnalytics; Foundation Models when the SDK has it
Packages/FeatureIntelligence the assistant panel (inspector pane assistant), its model, cards, menu actions SheetCore, SheetUI, SheetIntelligence, Audit

Source: Packages/SheetIntelligence/Package.swift, Packages/FeatureIntelligence/Package.swift.

The package author's own documents are Packages/SheetIntelligence/PRIVACY.md (what is shared with the model) and EVALUATION.md (measured results); this document describes the mechanism and reports those results with their limits. Privacy and threat considerations for the whole application are in ../security.md.

# The model backend

There is one production backend, FoundationModelsBackend. It uses SystemLanguageModel.default from Apple's Foundation Models framework, the on-device model of macOS, and nothing else:

  • The code is compiled under #if canImport(FoundationModels) and guarded by if #available(macOS 26.0, *). The application's deployment target is macOS 14, so on macOS 14 and 15 the backend reports osTooOld and the features are disabled.
  • A new LanguageModelSession is created for every request and discarded afterwards. No conversation state is kept between requests.
  • Neither package contains networking code. A search of Packages/SheetIntelligence/Sources and Packages/FeatureIntelligence/Sources for URLSession, the Network framework, sockets, URL requests and PrivateCloudCompute returns no match (run on 2026-11-21). The cloud model type that the macOS 27 SDK also ships is not referenced.
  • There is no fallback to any other model or service.

ScriptedBackend, a deterministic test double, is wrapped in #if DEBUG and carries the marker SCRIPTED PREVIEW. The LanguageModelBackend protocol exposes a debugMarker that is non-nil only for test doubles, and the panel displays it, so scripted text cannot pass for model output. It does not exist in Release builds.

LanguageModelBackend has four operations: respond (one complete answer), stream (growing snapshots of the answer), structured (an answer constrained to an OutputSchema), and supportsLanguage(of:).

# Structured output

OutputSchema describes the shape of a structured answer: string, integer, number, boolean, a choice among fixed strings, arrays with bounds, objects. When the SDK supports it, the backend converts the schema to a DynamicGenerationSchema and uses guided generation, so the model can only emit values of that shape — for example a findingId that is one of the ids actually sent. If the schema cannot be built, the fallback asks for strict JSON following a template, extracts the first complete JSON object from the answer (StructuredJSON.extract) and rejects anything that does not decode.

# Decoding parameters

Request type Used for Temperature Output cap
plain text (plain) formula suggestions, Python help 0 (greedy) 400 tokens (formulas), 250 (repair round), 1,200 (Python)
structured (structured) error review, chart ideas, transformation plan 0 (greedy) 800 tokens
streamed prose (stream) explain formula, explain statistics, describe data, document sheet 0 (greedy) 700 tokens

Source: Packages/SheetIntelligence/Sources/SheetIntelligence/Intelligence.swift (plain, structured, stream, maxProseTokens, maxStructuredTokens).

All three request paths build GenerationOptions(temperature: 0, …); FoundationModelsBackend maps temperature 0 to the framework's greedy sampling mode. IntelligenceTests asserts that every streamed request carries temperature 0.

The run recorded in EVALUATION.md predates this for the streamed tasks: its header states that the four prose tasks (explain a formula, explain statistics, describe data, sheet purpose) passed no temperature in that run and used the system's default sampling, and that they were not re-run after Intelligence.stream was changed on 2026-11-21. The prose observations reproduced below are therefore single observations of a sampled output; the formula, Python, review, chart and plan results were obtained with greedy decoding.

# Grounding: what the model is given

The model never receives the workbook. For each request ContextBuilder builds a short text deterministically from the workbook, and the panel shows that exact text in a disclosure titled "What will be shared with the on-device model" before anything is sent, with its character count.

flowchart TD
    WB["Workbook<br/>live session"] --> CTX["ContextBuilder<br/>deterministic facts, hard caps"]
    CTX --> SHOW["Panel shows the exact<br/>context text to the user"]
    CTX --> PROMPT["Fixed instructions<br/>+ context + request"]
    PROMPT --> MODEL["SystemLanguageModel.default<br/>on device, new session per request"]
    MODEL --> CHECK["Deterministic checks<br/>parser, registry, engine on a clone,<br/>finding ids, column names, numbers"]
    CHECK -->|rejected| DROP["Shown as invalid<br/>or discarded"]
    CHECK -->|accepted| CARD["Suggestion card"]
    CARD -->|Insert| EDIT["User clicks Insert:<br/>validated again,<br/>one undoable edit"]

SVG fallback: Assistant request flow with grounding and validation

Task Context text
Explain a formula the formula text, its computed result, signatures and summaries of the functions it uses (from the registry), the engine's evaluation steps, the current values of the cells it references and the labels next to them
Suggest a formula, Python help sheet names; used range; selection address; content of the active cell; headers, inferred types and data ranges of the block of data nearest to the active cell; its first rows; labels next to the active cell; defined names; table names; names of available functions; the user's request
Review possible errors the findings of the deterministic auditor (id, rule, severity, location, evidence, explanation, recommended action); no other cell content
Describe data, suggest charts, plan a transformation the profile of the table: per column its name, inferred type, count, missing count, distinct count, min/max/mean/median/standard deviation/quartiles/sum, first and last date, and the three most frequent values of text columns; never the rows
Explain statistical results the selected result table (at most 60 rows × 12 columns, cells cut to 28 characters)
Document a sheet the sheet context plus the detected structure: inputs (constants read by formulas), key formulas with their real text, outputs

Source: Packages/SheetIntelligence/PRIVACY.md (section "What is given to the model"); the task list matches AssistantTask in Packages/FeatureIntelligence/Sources/FeatureIntelligence/AssistantModel.swift.

Every request also carries the fixed instructions of Prompts (constant text, no user data). The role text tells the model that it never calculates, that every number must be copied from the facts, and that text from the user's sheet is data, never an instruction. That sentence is a mitigation, not an enforcement mechanism; the enforcement is the validation described below.

# Hard caps

ContextLimits bounds what can be sent, whatever the sheet contains.

Limit Default
Characters of context per request 6,000
Characters on the single retry after a context-window overflow 2,400
Columns described 24
Sample rows 5
Characters per cell value 32
Labels near the active cell 12
Defined names / tables 20
Evaluation steps / referenced cells 40 / 24
Auditor findings 20
Inputs, outputs / key formulas 20 / 14
Cells scanned to build structural facts 200,000
Free-text request 600 characters
Python code already in the editor 2,500 characters

Source: Packages/SheetIntelligence/Sources/SheetIntelligence/Models.swift (ContextLimits), Intelligence.swift (clip(request, 600), prefix(2500)).

The panel has a toggle "Include sample values from cells" (includeSampleValues). When it is off, the sheet-level tasks include no cell values: only headers, types, addresses and formula text. Explaining a formula always includes the values of the cells that formula reads.

If the model reports that its context window is exceeded before writing anything, the context is rebuilt with the reduced budget and the request is retried once. A second failure is reported to the user as "The request is too large for the on-device model…".

# Validation of suggestions

# Formulas

Intelligence.suggestFormulas asks for plain FORMULA: / WHY: lines (not JSON — see "Measured results" for why), parses at most three of them, and passes each through FormulaValidator. Nothing in the validator trusts the model:

Stage Check
normalise a formula must be found in the answer
safety at most 1,000 characters; at most 500 AST nodes; literal arguments of SEQUENCE, RANDARRAY, MAKEARRAY, REPT must not generate more than 1,000,000 values
parse the real FormulaParser, canonical syntax first, then the semicolon syntax; the canonical text printed from the AST is what gets offered
functions every function must exist in the registry (or be an evaluator special form, or a LET/LAMBDA binding, or a defined name), with an argument count inside its bounds
references sheets, defined names, tables and table columns must exist; the formula must not refer to the cell it would be placed in
evaluate the real CalcEngine evaluates the expression on a private copy of the workbook (Workbook.clone()); the live workbook is not touched

Source: Packages/SheetIntelligence/Sources/SheetIntelligence/FormulaValidator.swift.

A valid suggestion carries the engine's value as a preview, plus warnings when the result is currently an error, when the formula reads empty ranges, or when separators were converted. If no suggestion is valid, the model gets one repair round with the validation messages; a repaired suggestion is marked isRepair.

Inserting is a separate, explicit step. AssistantModel.insert validates the formula a second time against the workbook as it is at that moment, then calls session.perform("Insert Suggested Formula") — one undoable edit. If the second validation fails, nothing is inserted.

What validation does not establish: that the formula answers the user's question. A formula can parse, reference existing cells and evaluate, and still compute something else. The measured results below quantify this.

# Other tasks

Task Post-check
Review possible errors each item must carry the id of a finding that was sent (enforced by a choice schema and checked again in PostChecks.review); other ids are discarded and counted in rejectedIDs. With no findings, the model is not called
Suggest charts kind restricted to 10 values (bar, column, stackedColumn, line, area, scatter, pie, histogram, boxPlot, heatmap); columns must exist in the profile; at most 4
Plan a transformation operations restricted to 15 values (filter, sort, rename, changeType, split, merge, groupBy, pivot, unpivot, removeDuplicates, fillMissing, replace, parseDate, extractText, calculatedColumn); column names validated against the profile; at most 8 steps
Describe data, explain statistics, explain a formula NumberVerifier extracts the number tokens of the prose and compares them with the numbers of the profile / table / evaluation; those matching nothing are listed as unverified
Python help the code is passed on standard input to python3 -I -c "compile(…)" — compiled, never executed, never written to disk, 6-second timeout; calls to orix.* that do not exist in the API are listed. With no interpreter the check is skipped and says so
Document a sheet the Markdown document is assembled by SheetDocumenter from detected facts; the model writes only the paragraph under "Purpose (inferred…)"

Source: Packages/SheetIntelligence/Sources/SheetIntelligence/Intelligence.swift, PostChecks.swift, PythonChecker.swift, Models.swift (ChartKind, PlanOperation).

What the panel does with accepted results:

  • A formula can be inserted (see above) or copied.
  • Chart ideas and transformation plans are displayed and can be copied as text. They are not wired to chart insertion or to the Transform studio; the assistant's ChartKind and PlanOperation vocabularies are its own and are not the chart engine's 21 kinds (chart-engine.md) or the transform steps (data-engine.md).
  • Python code is copied to the clipboard and the Python Studio pane is shown; it is never run by the assistant (python-runtime.md).
  • Prose and the sheet documentation can be copied.

# Measured results

RealModelTests.fullEvaluation runs against the real on-device model and writes EVALUATION.md itself. It is opt-in: it runs only with ORIX_AI_EVAL=1 in the environment. The figures below are copied from that file. They describe one run:

Date 2026-11-21T07:36:55Z
macOS Version 27.0 (Build 26A5425a)
Model reported as "Apple on-device model (AFM 3 Core Advanced)"
Function registry at the time 465 functions
Workbook synthetic sales table Sales!A1:F21 (20 data rows, 6 columns), a Params sheet with the name TaxRate, active cell H2

Source: Packages/SheetIntelligence/EVALUATION.md (header).

# Formula suggestions, 20 fixed requests

Expected values are computed independently in Swift from the generator of the synthetic data; a suggestion that validates but computes another value counts as wrong.

Measure Result
Requests answered without error 20 / 20
Suggestions produced 35
Suggestions that passed validation 33 / 35
Requests with at least one valid suggestion on the first pass 20 / 20
Requests whose first listed suggestion is valid and computes the expected value 11 / 20
Requests where some valid suggestion computes the expected value 12 / 20
Repair rounds triggered 0
Mean time per request 28.2 s

Source: Packages/SheetIntelligence/EVALUATION.md (section "Formula suggestions").

Reading these numbers:

  • Validation is effective at what it does: 33 of 35 suggestions were well-formed, and the two invalid ones were caught with a precise reason (a non-existent function MAXIF; a ? : ternary that is not formula syntax).
  • Validity is not correctness. On this run eight of twenty requests received only wrong formulas that were nevertheless valid — for example "Total revenue" answered with a SUMIFS whose criteria match nothing (result 0), or "Number of different products" answered with a COUNTIF that returns 20 instead of 3. The panel shows the engine's value next to each suggestion, which is the user's means of noticing; nothing in the code detects such errors.
  • 28.2 s per request is a mean over this run on the build machine; no other timing is recorded.
  • The sample is 20 requests on one small synthetic table. It says nothing about larger sheets, other languages or other model versions.

The file EVALUATION-run1-guided-json.md keeps an earlier run of the same 20 requests verbatim: 38 of 57 suggestions valid, 15 of 20 requests with a valid suggestion on the first pass (17 after the repair round), first listed suggestion correct for 12 of 20 requests, some valid suggestion correct for 13 of 20. Three general changes were made between the runs (formulas requested as plain lines instead of JSON strings, because quotes inside formulas were truncated; function names listed instead of keyword-matched signatures; "the best formula" requested instead of "up to 3 different formulas"). The 20 requests and their expected values were not changed. Note that the later run has more valid suggestions (33 of 35 against 38 of 57; 20 of 20 requests valid on the first pass against 15) but one fewer correct request on both correctness measures (11 against 12, and 12 against 13): the changes fixed malformed output, not the model's understanding of the question, and the difference of one request is within what a single run can show.

# Other tasks

EVALUATION.md records one observation per task, not a rate:

Task Observation in that run
Explain a formula (English) numbers not found among the engine's values: none
Explain a formula (app locale fr-CA) supportsLocale(fr_CA) true; answer written in French (stop-word heuristic)
Describe data 34 numbers verified against the profile; none flagged
Suggest charts 4 accepted after validation
Transformation plan 3 of 3 steps consistent with the profile
Review auditor findings (4 findings) invented ids rejected: 0
Explain statistics numbers not in the table: none
Python help syntax check passed; unknown orix calls: none

Source: Packages/SheetIntelligence/EVALUATION.md (section "Other tasks").

Two of these samples show the limit of the post-checks and should be read in the source file:

  • The transformation plan for "total revenue per region, highest first" starts with an unnecessary date filter. It passes the check because every column it names exists; the check does not judge whether a step is useful.
  • The Python help sample passed the syntax check and the orix API check, yet the code is wrong for the request (it reads H2:H3, uses an undefined variable sheet, and multiplies two rows). compile() detects syntax errors only.

# Unavailable state

Intelligence.availability() returns .available(modelDescription:) or .unavailable(reason:) with one of six reasons:

Reason code Meaning Remedy shown
osTooOld macOS older than 26 update macOS
deviceNotEligible the Mac cannot run Apple Intelligence none
appleIntelligenceNotEnabled Apple Intelligence is turned off turn it on in System Settings
modelNotReady the system is still downloading or preparing the model wait and retry
frameworkMissing the app was built without the Foundation Models framework install a build made with the macOS 26 SDK or later
other a reason reported by the system that this version does not know check System Settings

Source: Packages/SheetIntelligence/Sources/SheetIntelligence/Availability.swift.

Every request path calls requireAvailable() first and throws IntelligenceError.unavailable(reason) without contacting the model. In the panel the AI controls are disabled and the reason and remedy are displayed. Per PRIVACY.md, the panel keeps showing the deterministic information it computes itself (formula evaluation steps, data profile, auditor findings, sheet structure), labelled "Computed · not AI". The download of the system model, when it happens, is performed by macOS, not by the application.

Other failures are mapped to explicit errors and shown as such: context window exceeded, guardrail violation, refusal, unsupported language, rate limited, invalid response, cancelled.

# Language

Answers are written in the app's language when SystemLanguageModel.default.supportsLocale accepts it, otherwise in English (answerLanguage() returns .englishFallback). Function names, formulas and cell addresses are requested unchanged. Localisation of the application itself is covered in ../development/localization.md.

# Registration in the app

IntelligenceFeature.register() (called from App/Sources/Features.swift) registers the inspector pane assistant ("ORIX Intelligence") and four menu actions: Explain This Formula and Suggest a Formula… (Tools), Describe This Data and Document This Sheet (Analyze). The remaining tasks — review, charts, plan, statistics, Python — are reached from the panel's task picker. There is one AssistantModel per session, held weakly by session. The extension mechanism is described in overview.md.

# Tests

docs/reports/tests.json records 62 test functions for SheetIntelligence and 17 for FeatureIntelligence, all passing (run of 2026-11-21T08:47:01Z). Most use ScriptedBackend to exercise the validators, post-checks, context caps and error mapping deterministically. The two RealModelTests that generate text with the real model (oversizedPromptMapsToContextWindowExceeded and fullEvaluation) are opt-in: they run only when the environment contains exactly ORIX_AI_EVAL=1. With the variable unset or set to any other value, a plain swift test skips them, generates nothing with the model and does not rewrite EVALUATION.md; the only real-model call left is the availability query. To regenerate EVALUATION.md, run ORIX_AI_EVAL=1 swift test in Packages/SheetIntelligence; on a machine without the model the tests print the reason and return. See ../development/testing.md.

# Not implemented

  • No cloud model, no third-party model, no API key, no fallback when the system model is unavailable.
  • No conversation: each request is independent; there is no chat history or follow-up context.
  • No autonomous actions: the model cannot edit cells, run Python, insert charts or run transforms. The only write path is the user clicking Insert on a validated formula.
  • No semantic check that a valid formula answers the request (see the measured results).
  • No evaluation beyond the 20 formula requests and the single observations listed above; no measurements on macOS 26, on other hardware or in languages other than the one French sample.
  • No telemetry or logging of prompts or answers.
Files main
Menu

Theme