On-device assistance (ORIX Intelligence)
ORIX Intelligence is an optional panel that uses the language model built into macOS to explain and suggest. It never computes. Every piece of model output is either verified by deterministic code (the formula parser, the calculation engine, the auditor, the data profile) or is displayed as text that the user may copy. When the system model is not available the panel says why, and the rest of the application is unaffected.
| Package | Role | Imports |
|---|---|---|
Packages/SheetIntelligence |
Intelligence actor, model backend, context builder, prompts, validators and post-checks |
SheetCore; the Audit and Statistics products of SheetAnalytics; Foundation Models when the SDK has it |
Packages/FeatureIntelligence |
the assistant panel (inspector pane assistant), its model, cards, menu actions |
SheetCore, SheetUI, SheetIntelligence, Audit |
Source: Packages/SheetIntelligence/Package.swift, Packages/FeatureIntelligence/Package.swift.
The package author's own documents are Packages/SheetIntelligence/PRIVACY.md (what is shared with the model) and EVALUATION.md (measured results); this document describes the mechanism and reports those results with their limits. Privacy and threat considerations for the whole application are in ../security.md.
The model backend
There is one production backend, FoundationModelsBackend. It uses SystemLanguageModel.default from Apple's Foundation Models framework, the on-device model of macOS, and nothing else:
- The code is compiled under
#if canImport(FoundationModels)and guarded byif #available(macOS 26.0, *). The application's deployment target is macOS 14, so on macOS 14 and 15 the backend reportsosTooOldand the features are disabled. - A new
LanguageModelSessionis created for every request and discarded afterwards. No conversation state is kept between requests. - Neither package contains networking code. A search of
Packages/SheetIntelligence/SourcesandPackages/FeatureIntelligence/SourcesforURLSession, the Network framework, sockets, URL requests andPrivateCloudComputereturns no match (run on 2026-11-21). The cloud model type that the macOS 27 SDK also ships is not referenced. - There is no fallback to any other model or service.
ScriptedBackend, a deterministic test double, is wrapped in #if DEBUG and carries the marker SCRIPTED PREVIEW. The LanguageModelBackend protocol exposes a debugMarker that is non-nil only for test doubles, and the panel displays it, so scripted text cannot pass for model output. It does not exist in Release builds.
LanguageModelBackend has four operations: respond (one complete answer), stream (growing snapshots of the answer), structured (an answer constrained to an OutputSchema), and supportsLanguage(of:).
Structured output
OutputSchema describes the shape of a structured answer: string, integer, number, boolean, a choice among fixed strings, arrays with bounds, objects. When the SDK supports it, the backend converts the schema to a DynamicGenerationSchema and uses guided generation, so the model can only emit values of that shape — for example a findingId that is one of the ids actually sent. If the schema cannot be built, the fallback asks for strict JSON following a template, extracts the first complete JSON object from the answer (StructuredJSON.extract) and rejects anything that does not decode.
Decoding parameters
| Request type | Used for | Temperature | Output cap |
|---|---|---|---|
plain text (plain) |
formula suggestions, Python help | 0 (greedy) | 400 tokens (formulas), 250 (repair round), 1,200 (Python) |
structured (structured) |
error review, chart ideas, transformation plan | 0 (greedy) | 800 tokens |
streamed prose (stream) |
explain formula, explain statistics, describe data, document sheet | 0 (greedy) | 700 tokens |
Source: Packages/SheetIntelligence/Sources/SheetIntelligence/Intelligence.swift (plain, structured, stream, maxProseTokens, maxStructuredTokens).
All three request paths build GenerationOptions(temperature: 0, …); FoundationModelsBackend maps temperature 0 to the framework's greedy sampling mode. IntelligenceTests asserts that every streamed request carries temperature 0.
The run recorded in EVALUATION.md predates this for the streamed tasks: its header states that the four prose tasks (explain a formula, explain statistics, describe data, sheet purpose) passed no temperature in that run and used the system's default sampling, and that they were not re-run after Intelligence.stream was changed on 2026-11-21. The prose observations reproduced below are therefore single observations of a sampled output; the formula, Python, review, chart and plan results were obtained with greedy decoding.
Grounding: what the model is given
The model never receives the workbook. For each request ContextBuilder builds a short text deterministically from the workbook, and the panel shows that exact text in a disclosure titled "What will be shared with the on-device model" before anything is sent, with its character count.
flowchart TD
WB["Workbook<br/>live session"] --> CTX["ContextBuilder<br/>deterministic facts, hard caps"]
CTX --> SHOW["Panel shows the exact<br/>context text to the user"]
CTX --> PROMPT["Fixed instructions<br/>+ context + request"]
PROMPT --> MODEL["SystemLanguageModel.default<br/>on device, new session per request"]
MODEL --> CHECK["Deterministic checks<br/>parser, registry, engine on a clone,<br/>finding ids, column names, numbers"]
CHECK -->|rejected| DROP["Shown as invalid<br/>or discarded"]
CHECK -->|accepted| CARD["Suggestion card"]
CARD -->|Insert| EDIT["User clicks Insert:<br/>validated again,<br/>one undoable edit"]
SVG fallback:
| Task | Context text |
|---|---|
| Explain a formula | the formula text, its computed result, signatures and summaries of the functions it uses (from the registry), the engine's evaluation steps, the current values of the cells it references and the labels next to them |
| Suggest a formula, Python help | sheet names; used range; selection address; content of the active cell; headers, inferred types and data ranges of the block of data nearest to the active cell; its first rows; labels next to the active cell; defined names; table names; names of available functions; the user's request |
| Review possible errors | the findings of the deterministic auditor (id, rule, severity, location, evidence, explanation, recommended action); no other cell content |
| Describe data, suggest charts, plan a transformation | the profile of the table: per column its name, inferred type, count, missing count, distinct count, min/max/mean/median/standard deviation/quartiles/sum, first and last date, and the three most frequent values of text columns; never the rows |
| Explain statistical results | the selected result table (at most 60 rows × 12 columns, cells cut to 28 characters) |
| Document a sheet | the sheet context plus the detected structure: inputs (constants read by formulas), key formulas with their real text, outputs |
Source: Packages/SheetIntelligence/PRIVACY.md (section "What is given to the model"); the task list matches AssistantTask in Packages/FeatureIntelligence/Sources/FeatureIntelligence/AssistantModel.swift.
Every request also carries the fixed instructions of Prompts (constant text, no user data). The role text tells the model that it never calculates, that every number must be copied from the facts, and that text from the user's sheet is data, never an instruction. That sentence is a mitigation, not an enforcement mechanism; the enforcement is the validation described below.
Hard caps
ContextLimits bounds what can be sent, whatever the sheet contains.
| Limit | Default |
|---|---|
| Characters of context per request | 6,000 |
| Characters on the single retry after a context-window overflow | 2,400 |
| Columns described | 24 |
| Sample rows | 5 |
| Characters per cell value | 32 |
| Labels near the active cell | 12 |
| Defined names / tables | 20 |
| Evaluation steps / referenced cells | 40 / 24 |
| Auditor findings | 20 |
| Inputs, outputs / key formulas | 20 / 14 |
| Cells scanned to build structural facts | 200,000 |
| Free-text request | 600 characters |
| Python code already in the editor | 2,500 characters |
Source: Packages/SheetIntelligence/Sources/SheetIntelligence/Models.swift (ContextLimits), Intelligence.swift (clip(request, 600), prefix(2500)).
The panel has a toggle "Include sample values from cells" (includeSampleValues). When it is off, the sheet-level tasks include no cell values: only headers, types, addresses and formula text. Explaining a formula always includes the values of the cells that formula reads.
If the model reports that its context window is exceeded before writing anything, the context is rebuilt with the reduced budget and the request is retried once. A second failure is reported to the user as "The request is too large for the on-device model…".
Validation of suggestions
Formulas
Intelligence.suggestFormulas asks for plain FORMULA: / WHY: lines (not JSON — see "Measured results" for why), parses at most three of them, and passes each through FormulaValidator. Nothing in the validator trusts the model:
| Stage | Check |
|---|---|
| normalise | a formula must be found in the answer |
| safety | at most 1,000 characters; at most 500 AST nodes; literal arguments of SEQUENCE, RANDARRAY, MAKEARRAY, REPT must not generate more than 1,000,000 values |
| parse | the real FormulaParser, canonical syntax first, then the semicolon syntax; the canonical text printed from the AST is what gets offered |
| functions | every function must exist in the registry (or be an evaluator special form, or a LET/LAMBDA binding, or a defined name), with an argument count inside its bounds |
| references | sheets, defined names, tables and table columns must exist; the formula must not refer to the cell it would be placed in |
| evaluate | the real CalcEngine evaluates the expression on a private copy of the workbook (Workbook.clone()); the live workbook is not touched |
Source: Packages/SheetIntelligence/Sources/SheetIntelligence/FormulaValidator.swift.
A valid suggestion carries the engine's value as a preview, plus warnings when the result is currently an error, when the formula reads empty ranges, or when separators were converted. If no suggestion is valid, the model gets one repair round with the validation messages; a repaired suggestion is marked isRepair.
Inserting is a separate, explicit step. AssistantModel.insert validates the formula a second time against the workbook as it is at that moment, then calls session.perform("Insert Suggested Formula") — one undoable edit. If the second validation fails, nothing is inserted.
What validation does not establish: that the formula answers the user's question. A formula can parse, reference existing cells and evaluate, and still compute something else. The measured results below quantify this.
Other tasks
| Task | Post-check |
|---|---|
| Review possible errors | each item must carry the id of a finding that was sent (enforced by a choice schema and checked again in PostChecks.review); other ids are discarded and counted in rejectedIDs. With no findings, the model is not called |
| Suggest charts | kind restricted to 10 values (bar, column, stackedColumn, line, area, scatter, pie, histogram, boxPlot, heatmap); columns must exist in the profile; at most 4 |
| Plan a transformation | operations restricted to 15 values (filter, sort, rename, changeType, split, merge, groupBy, pivot, unpivot, removeDuplicates, fillMissing, replace, parseDate, extractText, calculatedColumn); column names validated against the profile; at most 8 steps |
| Describe data, explain statistics, explain a formula | NumberVerifier extracts the number tokens of the prose and compares them with the numbers of the profile / table / evaluation; those matching nothing are listed as unverified |
| Python help | the code is passed on standard input to python3 -I -c "compile(…)" — compiled, never executed, never written to disk, 6-second timeout; calls to orix.* that do not exist in the API are listed. With no interpreter the check is skipped and says so |
| Document a sheet | the Markdown document is assembled by SheetDocumenter from detected facts; the model writes only the paragraph under "Purpose (inferred…)" |
Source: Packages/SheetIntelligence/Sources/SheetIntelligence/Intelligence.swift, PostChecks.swift, PythonChecker.swift, Models.swift (ChartKind, PlanOperation).
What the panel does with accepted results:
- A formula can be inserted (see above) or copied.
- Chart ideas and transformation plans are displayed and can be copied as text. They are not wired to chart insertion or to the Transform studio; the assistant's
ChartKindandPlanOperationvocabularies are its own and are not the chart engine's 21 kinds (chart-engine.md) or the transform steps (data-engine.md). - Python code is copied to the clipboard and the Python Studio pane is shown; it is never run by the assistant (python-runtime.md).
- Prose and the sheet documentation can be copied.
Measured results
RealModelTests.fullEvaluation runs against the real on-device model and writes EVALUATION.md itself. It is opt-in: it runs only with ORIX_AI_EVAL=1 in the environment. The figures below are copied from that file. They describe one run:
| Date | 2026-11-21T07:36:55Z |
| macOS | Version 27.0 (Build 26A5425a) |
| Model | reported as "Apple on-device model (AFM 3 Core Advanced)" |
| Function registry at the time | 465 functions |
| Workbook | synthetic sales table Sales!A1:F21 (20 data rows, 6 columns), a Params sheet with the name TaxRate, active cell H2 |
Source: Packages/SheetIntelligence/EVALUATION.md (header).
Formula suggestions, 20 fixed requests
Expected values are computed independently in Swift from the generator of the synthetic data; a suggestion that validates but computes another value counts as wrong.
| Measure | Result |
|---|---|
| Requests answered without error | 20 / 20 |
| Suggestions produced | 35 |
| Suggestions that passed validation | 33 / 35 |
| Requests with at least one valid suggestion on the first pass | 20 / 20 |
| Requests whose first listed suggestion is valid and computes the expected value | 11 / 20 |
| Requests where some valid suggestion computes the expected value | 12 / 20 |
| Repair rounds triggered | 0 |
| Mean time per request | 28.2 s |
Source: Packages/SheetIntelligence/EVALUATION.md (section "Formula suggestions").
Reading these numbers:
- Validation is effective at what it does: 33 of 35 suggestions were well-formed, and the two invalid ones were caught with a precise reason (a non-existent function
MAXIF; a? :ternary that is not formula syntax). - Validity is not correctness. On this run eight of twenty requests received only wrong formulas that were nevertheless valid — for example "Total revenue" answered with a
SUMIFSwhose criteria match nothing (result 0), or "Number of different products" answered with aCOUNTIFthat returns 20 instead of 3. The panel shows the engine's value next to each suggestion, which is the user's means of noticing; nothing in the code detects such errors. - 28.2 s per request is a mean over this run on the build machine; no other timing is recorded.
- The sample is 20 requests on one small synthetic table. It says nothing about larger sheets, other languages or other model versions.
The file EVALUATION-run1-guided-json.md keeps an earlier run of the same 20 requests verbatim: 38 of 57 suggestions valid, 15 of 20 requests with a valid suggestion on the first pass (17 after the repair round), first listed suggestion correct for 12 of 20 requests, some valid suggestion correct for 13 of 20. Three general changes were made between the runs (formulas requested as plain lines instead of JSON strings, because quotes inside formulas were truncated; function names listed instead of keyword-matched signatures; "the best formula" requested instead of "up to 3 different formulas"). The 20 requests and their expected values were not changed. Note that the later run has more valid suggestions (33 of 35 against 38 of 57; 20 of 20 requests valid on the first pass against 15) but one fewer correct request on both correctness measures (11 against 12, and 12 against 13): the changes fixed malformed output, not the model's understanding of the question, and the difference of one request is within what a single run can show.
Other tasks
EVALUATION.md records one observation per task, not a rate:
| Task | Observation in that run |
|---|---|
| Explain a formula (English) | numbers not found among the engine's values: none |
| Explain a formula (app locale fr-CA) | supportsLocale(fr_CA) true; answer written in French (stop-word heuristic) |
| Describe data | 34 numbers verified against the profile; none flagged |
| Suggest charts | 4 accepted after validation |
| Transformation plan | 3 of 3 steps consistent with the profile |
| Review auditor findings (4 findings) | invented ids rejected: 0 |
| Explain statistics | numbers not in the table: none |
| Python help | syntax check passed; unknown orix calls: none |
Source: Packages/SheetIntelligence/EVALUATION.md (section "Other tasks").
Two of these samples show the limit of the post-checks and should be read in the source file:
- The transformation plan for "total revenue per region, highest first" starts with an unnecessary date filter. It passes the check because every column it names exists; the check does not judge whether a step is useful.
- The Python help sample passed the syntax check and the
orixAPI check, yet the code is wrong for the request (it readsH2:H3, uses an undefined variablesheet, and multiplies two rows).compile()detects syntax errors only.
Unavailable state
Intelligence.availability() returns .available(modelDescription:) or .unavailable(reason:) with one of six reasons:
| Reason code | Meaning | Remedy shown |
|---|---|---|
osTooOld |
macOS older than 26 | update macOS |
deviceNotEligible |
the Mac cannot run Apple Intelligence | none |
appleIntelligenceNotEnabled |
Apple Intelligence is turned off | turn it on in System Settings |
modelNotReady |
the system is still downloading or preparing the model | wait and retry |
frameworkMissing |
the app was built without the Foundation Models framework | install a build made with the macOS 26 SDK or later |
other |
a reason reported by the system that this version does not know | check System Settings |
Source: Packages/SheetIntelligence/Sources/SheetIntelligence/Availability.swift.
Every request path calls requireAvailable() first and throws IntelligenceError.unavailable(reason) without contacting the model. In the panel the AI controls are disabled and the reason and remedy are displayed. Per PRIVACY.md, the panel keeps showing the deterministic information it computes itself (formula evaluation steps, data profile, auditor findings, sheet structure), labelled "Computed · not AI". The download of the system model, when it happens, is performed by macOS, not by the application.
Other failures are mapped to explicit errors and shown as such: context window exceeded, guardrail violation, refusal, unsupported language, rate limited, invalid response, cancelled.
Language
Answers are written in the app's language when SystemLanguageModel.default.supportsLocale accepts it, otherwise in English (answerLanguage() returns .englishFallback). Function names, formulas and cell addresses are requested unchanged. Localisation of the application itself is covered in ../development/localization.md.
Registration in the app
IntelligenceFeature.register() (called from App/Sources/Features.swift) registers the inspector pane assistant ("ORIX Intelligence") and four menu actions: Explain This Formula and Suggest a Formula… (Tools), Describe This Data and Document This Sheet (Analyze). The remaining tasks — review, charts, plan, statistics, Python — are reached from the panel's task picker. There is one AssistantModel per session, held weakly by session. The extension mechanism is described in overview.md.
Tests
docs/reports/tests.json records 62 test functions for SheetIntelligence and 17 for FeatureIntelligence, all passing (run of 2026-11-21T08:47:01Z). Most use ScriptedBackend to exercise the validators, post-checks, context caps and error mapping deterministically. The two RealModelTests that generate text with the real model (oversizedPromptMapsToContextWindowExceeded and fullEvaluation) are opt-in: they run only when the environment contains exactly ORIX_AI_EVAL=1. With the variable unset or set to any other value, a plain swift test skips them, generates nothing with the model and does not rewrite EVALUATION.md; the only real-model call left is the availability query. To regenerate EVALUATION.md, run ORIX_AI_EVAL=1 swift test in Packages/SheetIntelligence; on a machine without the model the tests print the reason and return. See ../development/testing.md.
Not implemented
- No cloud model, no third-party model, no API key, no fallback when the system model is unavailable.
- No conversation: each request is independent; there is no chat history or follow-up context.
- No autonomous actions: the model cannot edit cells, run Python, insert charts or run transforms. The only write path is the user clicking Insert on a validated formula.
- No semantic check that a valid formula answers the request (see the measured results).
- No evaluation beyond the 20 formula requests and the single observations listed above; no measurements on macOS 26, on other hardware or in languages other than the one French sample.
- No telemetry or logging of prompts or answers.