SPB Git

Repository Public

intelligence.md 22.1 KB 246 lines markdown

1# On-device assistance (ORIX Intelligence)23ORIX Intelligence is an optional panel that uses the language model built into macOS to *explain* and *suggest*. It never computes. Every piece of model output is either verified by deterministic code (the formula parser, the calculation engine, the auditor, the data profile) or is displayed as text that the user may copy. When the system model is not available the panel says why, and the rest of the application is unaffected.45| Package | Role | Imports |6|---|---|---|7| `Packages/SheetIntelligence` | `Intelligence` actor, model backend, context builder, prompts, validators and post-checks | `SheetCore`; the `Audit` and `Statistics` products of `SheetAnalytics`; Foundation Models when the SDK has it |8| `Packages/FeatureIntelligence` | the assistant panel (inspector pane `assistant`), its model, cards, menu actions | `SheetCore`, `SheetUI`, `SheetIntelligence`, `Audit` |910Source: `Packages/SheetIntelligence/Package.swift`, `Packages/FeatureIntelligence/Package.swift`.1112The package author's own documents are `Packages/SheetIntelligence/PRIVACY.md` (what is shared with the model) and `EVALUATION.md` (measured results); this document describes the mechanism and reports those results with their limits. Privacy and threat considerations for the whole application are in [../security.md](../security.md).1314## The model backend1516There is one production backend, `FoundationModelsBackend`. It uses `SystemLanguageModel.default` from Apple's Foundation Models framework, the on-device model of macOS, and nothing else:1718- The code is compiled under `#if canImport(FoundationModels)` and guarded by `if #available(macOS 26.0, *)`. The application's deployment target is macOS 14, so on macOS 14 and 15 the backend reports `osTooOld` and the features are disabled.19- A new `LanguageModelSession` is created for every request and discarded afterwards. No conversation state is kept between requests.20- Neither package contains networking code. A search of `Packages/SheetIntelligence/Sources` and `Packages/FeatureIntelligence/Sources` for `URLSession`, the Network framework, sockets, URL requests and `PrivateCloudCompute` returns no match (run on 2026-11-21). The cloud model type that the macOS 27 SDK also ships is not referenced.21- There is no fallback to any other model or service.2223`ScriptedBackend`, a deterministic test double, is wrapped in `#if DEBUG` and carries the marker `SCRIPTED PREVIEW`. The `LanguageModelBackend` protocol exposes a `debugMarker` that is non-nil only for test doubles, and the panel displays it, so scripted text cannot pass for model output. It does not exist in Release builds.2425`LanguageModelBackend` has four operations: `respond` (one complete answer), `stream` (growing snapshots of the answer), `structured` (an answer constrained to an `OutputSchema`), and `supportsLanguage(of:)`.2627### Structured output2829`OutputSchema` describes the shape of a structured answer: string, integer, number, boolean, a choice among fixed strings, arrays with bounds, objects. When the SDK supports it, the backend converts the schema to a `DynamicGenerationSchema` and uses guided generation, so the model can only emit values of that shape — for example a `findingId` that is one of the ids actually sent. If the schema cannot be built, the fallback asks for strict JSON following a template, extracts the first complete JSON object from the answer (`StructuredJSON.extract`) and rejects anything that does not decode.3031### Decoding parameters3233| Request type | Used for | Temperature | Output cap |34|---|---|---|---|35| plain text (`plain`) | formula suggestions, Python help | 0 (greedy) | 400 tokens (formulas), 250 (repair round), 1,200 (Python) |36| structured (`structured`) | error review, chart ideas, transformation plan | 0 (greedy) | 800 tokens |37| streamed prose (`stream`) | explain formula, explain statistics, describe data, document sheet | 0 (greedy) | 700 tokens |3839Source: `Packages/SheetIntelligence/Sources/SheetIntelligence/Intelligence.swift` (`plain`, `structured`, `stream`, `maxProseTokens`, `maxStructuredTokens`).4041All three request paths build `GenerationOptions(temperature: 0, …)`; `FoundationModelsBackend` maps temperature 0 to the framework's greedy sampling mode. `IntelligenceTests` asserts that every streamed request carries temperature 0.4243The run recorded in `EVALUATION.md` predates this for the streamed tasks: its header states that the four prose tasks (explain a formula, explain statistics, describe data, sheet purpose) passed no temperature in that run and used the system's default sampling, and that they were not re-run after `Intelligence.stream` was changed on 2026-11-21. The prose observations reproduced below are therefore single observations of a sampled output; the formula, Python, review, chart and plan results were obtained with greedy decoding.4445## Grounding: what the model is given4647The model never receives the workbook. For each request `ContextBuilder` builds a short text deterministically from the workbook, and the panel shows **that exact text** in a disclosure titled "What will be shared with the on-device model" before anything is sent, with its character count.4849<!-- diagram: intelligence-flow -->50```mermaid51flowchart TD52    WB["Workbook<br/>live session"] --> CTX["ContextBuilder<br/>deterministic facts, hard caps"]53    CTX --> SHOW["Panel shows the exact<br/>context text to the user"]54    CTX --> PROMPT["Fixed instructions<br/>+ context + request"]55    PROMPT --> MODEL["SystemLanguageModel.default<br/>on device, new session per request"]56    MODEL --> CHECK["Deterministic checks<br/>parser, registry, engine on a clone,<br/>finding ids, column names, numbers"]57    CHECK -->|rejected| DROP["Shown as invalid<br/>or discarded"]58    CHECK -->|accepted| CARD["Suggestion card"]59    CARD -->|Insert| EDIT["User clicks Insert:<br/>validated again,<br/>one undoable edit"]60```61SVG fallback: ![Assistant request flow with grounding and validation](../assets/diagrams/intelligence-flow.svg)6263| Task | Context text |64|---|---|65| Explain a formula | the formula text, its computed result, signatures and summaries of the functions it uses (from the registry), the engine's evaluation steps, the current values of the cells it references and the labels next to them |66| Suggest a formula, Python help | sheet names; used range; selection address; content of the active cell; headers, inferred types and data ranges of the block of data nearest to the active cell; its first rows; labels next to the active cell; defined names; table names; names of available functions; the user's request |67| Review possible errors | the findings of the deterministic auditor (id, rule, severity, location, evidence, explanation, recommended action); no other cell content |68| Describe data, suggest charts, plan a transformation | the **profile** of the table: per column its name, inferred type, count, missing count, distinct count, min/max/mean/median/standard deviation/quartiles/sum, first and last date, and the three most frequent values of text columns; never the rows |69| Explain statistical results | the selected result table (at most 60 rows × 12 columns, cells cut to 28 characters) |70| Document a sheet | the sheet context plus the detected structure: inputs (constants read by formulas), key formulas with their real text, outputs |7172Source: `Packages/SheetIntelligence/PRIVACY.md` (section "What is given to the model"); the task list matches `AssistantTask` in `Packages/FeatureIntelligence/Sources/FeatureIntelligence/AssistantModel.swift`.7374Every request also carries the fixed instructions of `Prompts` (constant text, no user data). The role text tells the model that it never calculates, that every number must be copied from the facts, and that text from the user's sheet is data, never an instruction. That sentence is a mitigation, not an enforcement mechanism; the enforcement is the validation described below.7576### Hard caps7778`ContextLimits` bounds what can be sent, whatever the sheet contains.7980| Limit | Default |81|---|---|82| Characters of context per request | 6,000 |83| Characters on the single retry after a context-window overflow | 2,400 |84| Columns described | 24 |85| Sample rows | 5 |86| Characters per cell value | 32 |87| Labels near the active cell | 12 |88| Defined names / tables | 20 |89| Evaluation steps / referenced cells | 40 / 24 |90| Auditor findings | 20 |91| Inputs, outputs / key formulas | 20 / 14 |92| Cells scanned to build structural facts | 200,000 |93| Free-text request | 600 characters |94| Python code already in the editor | 2,500 characters |9596Source: `Packages/SheetIntelligence/Sources/SheetIntelligence/Models.swift` (`ContextLimits`), `Intelligence.swift` (`clip(request, 600)`, `prefix(2500)`).9798The panel has a toggle "Include sample values from cells" (`includeSampleValues`). When it is off, the sheet-level tasks include no cell values: only headers, types, addresses and formula text. Explaining a formula always includes the values of the cells that formula reads.99100If the model reports that its context window is exceeded before writing anything, the context is rebuilt with the reduced budget and the request is retried once. A second failure is reported to the user as "The request is too large for the on-device model…".101102## Validation of suggestions103104### Formulas105106`Intelligence.suggestFormulas` asks for plain `FORMULA:` / `WHY:` lines (not JSON — see "Measured results" for why), parses at most three of them, and passes each through `FormulaValidator`. Nothing in the validator trusts the model:107108| Stage | Check |109|---|---|110| normalise | a formula must be found in the answer |111| safety | at most 1,000 characters; at most 500 AST nodes; literal arguments of `SEQUENCE`, `RANDARRAY`, `MAKEARRAY`, `REPT` must not generate more than 1,000,000 values |112| parse | the real `FormulaParser`, canonical syntax first, then the semicolon syntax; the canonical text printed from the AST is what gets offered |113| functions | every function must exist in the registry (or be an evaluator special form, or a LET/LAMBDA binding, or a defined name), with an argument count inside its bounds |114| references | sheets, defined names, tables and table columns must exist; the formula must not refer to the cell it would be placed in |115| evaluate | the real `CalcEngine` evaluates the expression **on a private copy** of the workbook (`Workbook.clone()`); the live workbook is not touched |116117Source: `Packages/SheetIntelligence/Sources/SheetIntelligence/FormulaValidator.swift`.118119A valid suggestion carries the engine's value as a preview, plus warnings when the result is currently an error, when the formula reads empty ranges, or when separators were converted. If no suggestion is valid, the model gets **one** repair round with the validation messages; a repaired suggestion is marked `isRepair`.120121Inserting is a separate, explicit step. `AssistantModel.insert` validates the formula a second time against the workbook as it is at that moment, then calls `session.perform("Insert Suggested Formula")` — one undoable edit. If the second validation fails, nothing is inserted.122123**What validation does not establish:** that the formula answers the user's question. A formula can parse, reference existing cells and evaluate, and still compute something else. The measured results below quantify this.124125### Other tasks126127| Task | Post-check |128|---|---|129| Review possible errors | each item must carry the id of a finding that was sent (enforced by a `choice` schema and checked again in `PostChecks.review`); other ids are discarded and counted in `rejectedIDs`. With no findings, the model is not called |130| Suggest charts | kind restricted to 10 values (`bar`, `column`, `stackedColumn`, `line`, `area`, `scatter`, `pie`, `histogram`, `boxPlot`, `heatmap`); columns must exist in the profile; at most 4 |131| Plan a transformation | operations restricted to 15 values (`filter`, `sort`, `rename`, `changeType`, `split`, `merge`, `groupBy`, `pivot`, `unpivot`, `removeDuplicates`, `fillMissing`, `replace`, `parseDate`, `extractText`, `calculatedColumn`); column names validated against the profile; at most 8 steps |132| Describe data, explain statistics, explain a formula | `NumberVerifier` extracts the number tokens of the prose and compares them with the numbers of the profile / table / evaluation; those matching nothing are listed as *unverified* |133| Python help | the code is passed on standard input to `python3 -I -c "compile(…)"` — compiled, never executed, never written to disk, 6-second timeout; calls to `orix.*` that do not exist in the API are listed. With no interpreter the check is skipped and says so |134| Document a sheet | the Markdown document is assembled by `SheetDocumenter` from detected facts; the model writes only the paragraph under "Purpose (inferred…)" |135136Source: `Packages/SheetIntelligence/Sources/SheetIntelligence/Intelligence.swift`, `PostChecks.swift`, `PythonChecker.swift`, `Models.swift` (`ChartKind`, `PlanOperation`).137138What the panel does with accepted results:139140- A formula can be inserted (see above) or copied.141- Chart ideas and transformation plans are **displayed and can be copied as text**. They are not wired to chart insertion or to the Transform studio; the assistant's `ChartKind` and `PlanOperation` vocabularies are its own and are not the chart engine's 21 kinds ([chart-engine.md](chart-engine.md)) or the transform steps ([data-engine.md](data-engine.md)).142- Python code is copied to the clipboard and the Python Studio pane is shown; it is never run by the assistant ([python-runtime.md](python-runtime.md)).143- Prose and the sheet documentation can be copied.144145## Measured results146147`RealModelTests.fullEvaluation` runs against the real on-device model and writes `EVALUATION.md` itself. It is opt-in: it runs only with `ORIX_AI_EVAL=1` in the environment. The figures below are copied from that file. They describe **one run**:148149| | |150|---|---|151| Date | 2026-11-21T07:36:55Z |152| macOS | Version 27.0 (Build 26A5425a) |153| Model | reported as "Apple on-device model (AFM 3 Core Advanced)" |154| Function registry at the time | 465 functions |155| Workbook | synthetic sales table `Sales!A1:F21` (20 data rows, 6 columns), a `Params` sheet with the name `TaxRate`, active cell `H2` |156157Source: `Packages/SheetIntelligence/EVALUATION.md` (header).158159### Formula suggestions, 20 fixed requests160161Expected values are computed independently in Swift from the generator of the synthetic data; a suggestion that validates but computes another value counts as wrong.162163| Measure | Result |164|---|---|165| Requests answered without error | 20 / 20 |166| Suggestions produced | 35 |167| Suggestions that passed validation | 33 / 35 |168| Requests with at least one valid suggestion on the first pass | 20 / 20 |169| Requests whose first listed suggestion is valid **and computes the expected value** | 11 / 20 |170| Requests where some valid suggestion computes the expected value | 12 / 20 |171| Repair rounds triggered | 0 |172| Mean time per request | 28.2 s |173174Source: `Packages/SheetIntelligence/EVALUATION.md` (section "Formula suggestions").175176Reading these numbers:177178- Validation is effective at what it does: 33 of 35 suggestions were well-formed, and the two invalid ones were caught with a precise reason (a non-existent function `MAXIF`; a `? :` ternary that is not formula syntax).179- Validity is not correctness. On this run **eight of twenty requests received only wrong formulas** that were nevertheless valid — for example "Total revenue" answered with a `SUMIFS` whose criteria match nothing (result 0), or "Number of different products" answered with a `COUNTIF` that returns 20 instead of 3. The panel shows the engine's value next to each suggestion, which is the user's means of noticing; nothing in the code detects such errors.180- 28.2 s per request is a mean over this run on the build machine; no other timing is recorded.181- The sample is 20 requests on one small synthetic table. It says nothing about larger sheets, other languages or other model versions.182183The file `EVALUATION-run1-guided-json.md` keeps an earlier run of the same 20 requests verbatim: 38 of 57 suggestions valid, 15 of 20 requests with a valid suggestion on the first pass (17 after the repair round), first listed suggestion correct for 12 of 20 requests, some valid suggestion correct for 13 of 20. Three general changes were made between the runs (formulas requested as plain lines instead of JSON strings, because quotes inside formulas were truncated; function names listed instead of keyword-matched signatures; "the best formula" requested instead of "up to 3 different formulas"). The 20 requests and their expected values were not changed. Note that the later run has *more valid* suggestions (33 of 35 against 38 of 57; 20 of 20 requests valid on the first pass against 15) but *one fewer correct request* on both correctness measures (11 against 12, and 12 against 13): the changes fixed malformed output, not the model's understanding of the question, and the difference of one request is within what a single run can show.184185### Other tasks186187`EVALUATION.md` records **one observation per task**, not a rate:188189| Task | Observation in that run |190|---|---|191| Explain a formula (English) | numbers not found among the engine's values: none |192| Explain a formula (app locale fr-CA) | `supportsLocale(fr_CA)` true; answer written in French (stop-word heuristic) |193| Describe data | 34 numbers verified against the profile; none flagged |194| Suggest charts | 4 accepted after validation |195| Transformation plan | 3 of 3 steps consistent with the profile |196| Review auditor findings (4 findings) | invented ids rejected: 0 |197| Explain statistics | numbers not in the table: none |198| Python help | syntax check passed; unknown `orix` calls: none |199200Source: `Packages/SheetIntelligence/EVALUATION.md` (section "Other tasks").201202Two of these samples show the limit of the post-checks and should be read in the source file:203204- The **transformation plan** for "total revenue per region, highest first" starts with an unnecessary date filter. It passes the check because every column it names exists; the check does not judge whether a step is useful.205- The **Python help** sample passed the syntax check and the `orix` API check, yet the code is wrong for the request (it reads `H2:H3`, uses an undefined variable `sheet`, and multiplies two rows). `compile()` detects syntax errors only.206207## Unavailable state208209`Intelligence.availability()` returns `.available(modelDescription:)` or `.unavailable(reason:)` with one of six reasons:210211| Reason code | Meaning | Remedy shown |212|---|---|---|213| `osTooOld` | macOS older than 26 | update macOS |214| `deviceNotEligible` | the Mac cannot run Apple Intelligence | none |215| `appleIntelligenceNotEnabled` | Apple Intelligence is turned off | turn it on in System Settings |216| `modelNotReady` | the system is still downloading or preparing the model | wait and retry |217| `frameworkMissing` | the app was built without the Foundation Models framework | install a build made with the macOS 26 SDK or later |218| `other` | a reason reported by the system that this version does not know | check System Settings |219220Source: `Packages/SheetIntelligence/Sources/SheetIntelligence/Availability.swift`.221222Every request path calls `requireAvailable()` first and throws `IntelligenceError.unavailable(reason)` without contacting the model. In the panel the AI controls are disabled and the reason and remedy are displayed. Per `PRIVACY.md`, the panel keeps showing the deterministic information it computes itself (formula evaluation steps, data profile, auditor findings, sheet structure), labelled "Computed · not AI". The download of the system model, when it happens, is performed by macOS, not by the application.223224Other failures are mapped to explicit errors and shown as such: context window exceeded, guardrail violation, refusal, unsupported language, rate limited, invalid response, cancelled.225226### Language227228Answers are written in the app's language when `SystemLanguageModel.default.supportsLocale` accepts it, otherwise in English (`answerLanguage()` returns `.englishFallback`). Function names, formulas and cell addresses are requested unchanged. Localisation of the application itself is covered in [../development/localization.md](../development/localization.md).229230## Registration in the app231232`IntelligenceFeature.register()` (called from `App/Sources/Features.swift`) registers the inspector pane `assistant` ("ORIX Intelligence") and four menu actions: Explain This Formula and Suggest a Formula… (Tools), Describe This Data and Document This Sheet (Analyze). The remaining tasks — review, charts, plan, statistics, Python — are reached from the panel's task picker. There is one `AssistantModel` per session, held weakly by session. The extension mechanism is described in [overview.md](overview.md).233234## Tests235236`docs/reports/tests.json` records 62 test functions for `SheetIntelligence` and 17 for `FeatureIntelligence`, all passing (run of 2026-11-21T08:47:01Z). Most use `ScriptedBackend` to exercise the validators, post-checks, context caps and error mapping deterministically. The two `RealModelTests` that generate text with the real model (`oversizedPromptMapsToContextWindowExceeded` and `fullEvaluation`) are opt-in: they run only when the environment contains exactly `ORIX_AI_EVAL=1`. With the variable unset or set to any other value, a plain `swift test` skips them, generates nothing with the model and does not rewrite `EVALUATION.md`; the only real-model call left is the availability query. To regenerate `EVALUATION.md`, run `ORIX_AI_EVAL=1 swift test` in `Packages/SheetIntelligence`; on a machine without the model the tests print the reason and return. See [../development/testing.md](../development/testing.md).237238## Not implemented239240- No cloud model, no third-party model, no API key, no fallback when the system model is unavailable.241- No conversation: each request is independent; there is no chat history or follow-up context.242- No autonomous actions: the model cannot edit cells, run Python, insert charts or run transforms. The only write path is the user clicking Insert on a validated formula.243- No semantic check that a valid formula answers the request (see the measured results).244- No evaluation beyond the 20 formula requests and the single observations listed above; no measurements on macOS 26, on other hardware or in languages other than the one French sample.245- No telemetry or logging of prompts or answers.246
Files main
Menu

Theme