Testing
The test strategy by layer, how to run the suites, where the fixtures come from, which tests are gated, and what the automated tests do not cover.
Related documents: setup.md, performance.md, accessibility.md, localization.md, ../mathematics/reference-tests.md (the reference sources and their case counts), ../release/checklist.md.
1. Where the numbers are
Test counts and pass/fail status change with every commit. The authoritative records are
docs/reports/tests.json, written by Scripts/test-all.sh (see section 3), and
docs/reports/app-tests.json for the app-hosted bundle. The table at the end of this document
(section 8) is copied from those two files: 22 packages, 1,270 test functions,
no failed package, plus 9 app-hosted tests with no failure.
2. Strategy by layer
flowchart TD
A["App integration tests<br/>XCTest hosted in the app<br/>App/Tests"] --> B["Feature packages<br/>view models and adapters<br/>Feature*/Tests"]
B --> C["UI kit<br/>session, layout, accessibility tree, contrast<br/>SheetUI/Tests"]
B --> D["Domain libraries<br/>SheetAnalytics, SheetData, SheetCharts,<br/>SheetPython, SheetIntelligence, SheetIO"]
B --> F["Function libraries<br/>FunctionsNumeric, FunctionsText, FunctionsLookup"]
C --> E["Core<br/>SheetCore, SheetFormat, SheetEngine"]
D --> E
F --> E
SVG fallback:
Each box is tested by its own suite; an arrow means "is built on". The domain and function libraries are the two layers whose expected values come from external reference programs (see ../mathematics/reference-tests.md).
| Layer | Packages | What the tests are | External reference |
|---|---|---|---|
| Core model and engine | SheetCore, SheetEngine |
hand-written expectations: addresses, parser, semantics, recalculation, editing with undo, input parsing, a performance smoke test; cross-library model and registry checks | none |
| Number formats | SheetFormat |
LibreOffice fixture replay with named deviation rules, hand-written edge cases, a fuzz test, a micro-benchmark suite | LibreOffice |
| Worksheet functions | FunctionsNumeric, FunctionsText, FunctionsLookup |
fixture replay (one parameterised test per fixture file), Microsoft documentation examples, hand-written edge cases, a test that fails when a registered function has no reference | LibreOffice, SciPy, mpmath, NumPy, statsmodels, Python decimal, Excel-computed public data |
| File formats | SheetIO |
read and write tests per format, ZIP and XML hardening (hostile archives, fuzzing of truncated and corrupted files), native-format integrity, interoperability with other programs, gated performance tests | openpyxl, pandas, LibreOffice |
| Statistics, finance, what-if, audit | SheetAnalytics (targets StatisticsTests, FinanceModelsTests, WhatIfTests, AuditTests, shared TestSupport) |
fixture replay against Python libraries; exact hand-derived expectations for the deterministic analyses | SciPy, NumPy, pandas, statsmodels, scikit-learn, arch, Python decimal |
| Data engine | SheetData |
engine, import, transform, pivot and sheet-bridge tests on seeded synthetic files | pandas (no DuckDB on the reference side) |
| Charts | SheetCharts |
numeric algorithms against Python references; every chart kind × difficult data rendered to SVG (parsed back with XMLParser), PDF and PNG; structural checks on the display list; vector-PDF heuristics; JSON round trips |
NumPy, SciPy, matplotlib |
| Python runtime | SheetPython |
execution, API, discovery, records and security tests that run real interpreters; each sandbox denial is asserted, and the same probe is asserted to succeed without the sandbox | the interpreters found on the Mac |
| Intelligence | SheetIntelligence |
context building, validators and post-checks with a fake backend; RealModelTests against the on-device model, opt-in (ORIX_AI_EVAL=1, section 5) |
the on-device model (measurement, not a reference) |
| UI kit | SheetUI |
AxisLayoutTests, SessionTests (edit / commit / undo, navigation, merge-aware selection, pointing, find, background recalculation, print pagination), ContrastTests, LocalizationTests, AccessibilityTreeTests |
WCAG 2.1 contrast formula |
| Feature panels | FeatureAudit, FeatureCharts, FeatureData, FeatureAnalysis, FeaturePython, FeatureIntelligence, FeatureAutomation |
view models, providers and adapters driven without a window; localization completeness; integration with a real WorkbookSession |
— |
| Application | App/Tests |
XCTest bundle hosted inside the running app: real documents, windows and grid, synthesized key and mouse events | — |
| Templates | SheetTemplates |
structure of each template, key outputs against expected.json, save and reload, LibreOffice recalculation of every template, thumbnails (section 7) |
numpy / scipy, LibreOffice |
| Automation engine | SheetAutomation |
workflow model, engine, steps, outputs, template merge, validation, the on-disk store, the orix-sheet command line (section 7) |
— |
Source: the test files under Packages/*/Tests and App/Tests (file and suite names), Packages/SheetCharts/README.md ("Tests"), Packages/SheetPython/SECURITY.md, Packages/SheetAnalytics/VALIDATION.md, Packages/SheetIO/COMPATIBILITY.md.
The packages use Swift Testing (@Test, @Suite, #expect); only the app-hosted bundle uses
XCTest (final class AppIntegrationTests: XCTestCase).
3. Running the tests
Everything, with a report
Scripts/test-all.sh # every package, sequentially
Scripts/test-all.sh SheetCore SheetIO # only the packages named
Scripts/build.sh test # same as Scripts/test-all.shWhat Scripts/test-all.sh does, read from the script:
- For each package, in this order —
SheetCore SheetFormat FunctionsNumeric FunctionsText FunctionsLookup SheetEngine SheetIO SheetCharts SheetData SheetAnalytics SheetPython SheetIntelligence SheetAutomation SheetTemplates SheetUI FeatureAudit FeatureCharts FeatureData FeatureAnalysis FeaturePython FeatureIntelligence FeatureAutomation— it runsORIX_AI_EVAL=0 swift test(a debug build) and stores the output, stripped of colour codes, inbuild/test-logs/<Package>.log. A package without aPackage.swiftis skipped. - The test count is the sum of every
Test run with N testsline (Swift Testing; a package may have several test targets) plus the lastExecuted N testsline (XCTest). A parameterised Swift Testing function counts once, whatever the number of arguments; a fixture file with thousands of cases is therefore one test. - A package is
passedwhen every such line contains "passed" and the log contains neithererror:norsignal code; otherwise it isFAILED. A failing package does not stop the others. - It prints one line per package (name, tests, status, seconds) and a total, and writes
docs/reports/tests.json:{"date", "total", "failedPackages", "packages": [{"package", "tests", "status", "seconds"}]}. - The exit status is 1 when at least one package failed.
The header comment of the script says it writes tests.json "+ a table"; the table is printed on
standard output only, it is not written to a file.
The on-device model evaluation is opt-in (ORIX_AI_EVAL=1, section 5); any other value, including the
0 the script sets, leaves it skipped. It takes minutes (mean 28.2 s per request over 20 requests in
Packages/SheetIntelligence/EVALUATION.md).
One package, one test
cd Packages/FunctionsLookup && swift test
cd Packages/SheetCore && swift test --filter RecalcTests
cd Packages/SheetIO && swift test --filter HardeningTestsUse --scratch-path <dir> when another build of the same package is running.
The app-hosted tests
project.yml declares the ORIXSheetTests bundle (sources in App/Tests, TEST_HOST = the Debug app)
and adds it to the test action of the ORIXSheet scheme, which passes -ApplePersistenceIgnoreState YES
and -ORIXTesting YES to the host app. No script in Scripts/ runs it: Scripts/build.sh test and
Scripts/test-all.sh cover the packages only. The command is:
Scripts/build.sh generate
xcodebuild test -project ORIXSheet.xcodeproj -scheme ORIXSheet -destination 'platform=macOS,arch=arm64'The recorded result is in docs/reports/app-tests.json: 9 tests, 0 failures
(2026-11-21T08:48:35Z). It is a separate file; the app-hosted tests are not part of
docs/reports/tests.json.
AppIntegrationTests.swift contains nine tests: features registered; typing formulas in the grid and
undo; mouse selection and redraw; native save and reopen is lossless; foreign files are imported as
untitled and export is preflighted; corrupted and unsupported files fail cleanly; printing produces a
paginated vector PDF; the menu bar has no duplicate shortcuts; a large sheet (50 000 rows × 4 columns,
100 000 formulas) recalculates in the background and repaints under a bound.
4. Fixtures and generators
Reference values are committed; swift test needs neither Python nor LibreOffice to replay them.
| Package | Fixtures (Tests/<Target>/Fixtures/) |
Generators (Tests/<Target>/Generators/) |
|---|---|---|
| FunctionsNumeric | 24 JSON files (lo_*.json, scipy_*.json, numpy_descriptive.json, statsmodels_regression.json, decimal_rounding.json, cmath_complex.json), 1.5 MiB |
gen_libreoffice.py, gen_scipy.py, common.py, data.py, overrides.py, coverage.py |
| FunctionsText | function_references.json, 0.8 MiB |
generate_function_fixtures.py, lo_recalc.py |
| FunctionsLookup | excel_reference.json, financial.json, securities.json, lookup_lo.json, arrays.json, 9.6 MiB |
gen_lookup.py, gen_arrays.py, gen_financial.py, gen_securities.py, gen_excel_reference.py, lo_harness.py, pyref.py |
| SheetFormat | libreoffice_formats.json, 0.2 MiB |
generate_format_fixtures.py, lo_recalc.py |
| SheetAnalytics | 7 JSON files for StatisticsTests, finance.json for FinanceModelsTests, 1.3 MiB |
gen_core.py, gen_descriptive_inference.py, gen_regression.py, gen_timeseries.py, gen_multivariate.py, gen_resampling.py, gen_tukey.py, gen_finance.py, gen_common.py |
| SheetData | 14 files: CSV dialects, Parquet, JSON Lines, SQLite, Arrow, expected.json, 1.9 MiB |
generate_fixtures.py |
| SheetCharts | reference.json, colormaps.json, 0.2 MiB |
gen_fixtures.py, gen_colormaps.py |
| SheetIO | 7 files: XLSX from openpyxl and LibreOffice, an ODS, a raw XLSX, a macro workbook, 0.1 MiB | gen_openpyxl_corpus.py, gen_raw_xlsx.py, gen_libreoffice_corpus.sh, dump_xlsx.py |
| SheetTemplates | expected.json |
gen_expected.py (reads inputs.json) |
Source: find Packages -path '*Fixtures*' -type f and find Packages -path '*/Generators/*', sizes summed with stat, run on 2026-11-21.
The other packages have no fixture directory: their tests build their inputs in code (synthetic workbooks, temporary files).
How each corpus is built, what the references are and how to regenerate them is described in ../mathematics/reference-tests.md. Two rules apply to anyone adding a function or a format:
- add reference cases to the generator, not expected values copied from the implementation;
- when a reference program is known to deviate from Excel's documented behaviour, record the Excel value
and the reference's value with a reason (the
overrides.py,excelsource tag andExcelDeviations.swiftmechanisms), so that the deviation stays visible.
5. Gated and conditional tests
Opt-in by environment variable
| Variable | Package / test | Effect |
|---|---|---|
ORIX_AI_EVAL=1 |
SheetIntelligence, RealModelTests.oversizedPromptMapsToContextWindowExceeded and RealModelTests.fullEvaluation |
enables the two tests that generate text with the on-device model; fullEvaluation runs the 20 requests and rewrites Packages/SheetIntelligence/EVALUATION.md. With the variable unset or set to anything else (0 included) both are reported as skipped, the model generates nothing and the file is not rewritten |
ORIX_BENCH=1 |
SheetIO, PerformanceTests |
enables the 1,000,000-cell and 5,000,000-cell runs: ORIX_BENCH=1 swift test -c release -Xswiftc -enable-testing --filter PerformanceTests |
ORIX_DATA_LARGE_TESTS=1 |
SheetData, millionRowPipeline |
enables the 1,000,000-row end-to-end test: ORIX_DATA_LARGE_TESTS=1 swift test --filter millionRow |
FN_DUMP=<dir> |
FunctionsNumeric, ReferenceFixtureTests |
writes every mismatch of each fixture to <dir>/<fixture>.txt (to review a regenerated corpus) |
ORIX_TEMPLATES_SKIP_LO |
SheetTemplates, LibreOfficeCrossCheck |
skips the LibreOffice recalculation of the templates |
ORIX_TEMPLATES_INPUTS=<file> |
SheetTemplates, exportGeneratorInputs |
writes the inputs read by gen_expected.py |
ORIX_TEMPLATES_DUMP=<file> |
SheetTemplates, dumpNamedValues |
debug aid: the value of every defined name of every template |
ORIX_TEMPLATES_THUMBS=<dir> |
SheetTemplates, Thumbnails |
also writes the thumbnail PNGs |
ORIX_TEMPLATES_NO_FEATURES, ORIX_TEMPLATES_NO_PREVIEW |
SheetTemplates/Package.swift |
build the package without the sibling feature packages / without the preview harness |
ORIX_SHEET_SUPPORT_DIR |
SheetPython, SheetAutomation |
points the application-support directory at a temporary location |
Source: grep -rn 'ProcessInfo.processInfo.environment' Packages --include='*.swift' and the files it lists, run on 2026-11-21.
Disabled when a tool or a permission is missing
These tests are skipped, not failed, when their precondition is not met. A green run on a machine without the tools has therefore verified less than a green run on the build machine.
| Condition | Tests affected |
|---|---|
/opt/homebrew/bin/python3 is not executable (Tools.hasPython) |
SheetIO: three InteropTests tests and one CSVTests test (the checks that run openpyxl and pandas on files SheetIO wrote) |
/opt/homebrew/bin/soffice is not executable (Tools.hasLibreOffice) |
SheetIO: one InteropTests test and one ODSTests test; SheetTemplates: the LibreOffice cross-check returns early |
No Python ≥ 3.9 interpreter found (Interp.any) |
most of SheetPython (execution, security, discovery, API) and four FeaturePython integration tests |
No interpreter with pandas + numpy + matplotlib (Interp.scientific) |
the DataFrame, plotting and large-data tests of SheetPython and FeaturePython |
No interpreter without pandas (Interp.bare, normally /usr/bin/python3) |
one SheetPython API test |
| polars not installed | one SheetPython API test |
The on-device model is unavailable (with ORIX_AI_EVAL=1) |
the two opt-in RealModelTests return early after printing the reason; EVALUATION.md is not written |
The test runner is not trusted for the accessibility API (AXIsProcessTrusted()) |
the whole AccessibilityTreeTests suite of SheetUI |
Source: grep -rn '\.enabled(' Packages/*/Tests, Packages/SheetIO/Tests/SheetIOTests/TestSupport.swift, Packages/SheetPython/Tests/SheetPythonTests/Support.swift, Packages/SheetIntelligence/Tests/SheetIntelligenceTests/RealModelTests.swift, Packages/SheetUI/Tests/SheetUITests/AccessibilityTests.swift.
Note on RealModelTests: availabilityIsReported always runs. It asks the system whether the model is
available and generates nothing; that query is the only contact with the real backend in a plain
swift test or a Scripts/test-all.sh run.
6. UI and automation testing
What is automated
- Session-level tests without a window (
SheetUI/SessionTests): theWorkbookSessionis the main actor object behind every window; editing, committing, undo, navigation, selection across merged cells, formula pointing, the find model, background recalculation and print pagination are tested on it directly. - Accessibility tree of a real view (
SheetUI/AccessibilityTreeTests): aWorkbookViewis put in an offscreen window and read through the system accessibility API; the numbers found (elements, controls, controls without a label) are reported in accessibility.md. - Contrast (
SheetUI/ContrastTests): every text / background pair of the design tokens is measured in light and dark and the test fails under 4.5:1 for text and 3:1 for glyphs and structural lines. - Feature view models (
Feature*/Tests): each panel exposes its state as a model that the tests drive without rendering;FeatureDataadds aDataAutomationentry point used by its tests and by thedata-previewbenchmark. - App-hosted integration (
App/Tests): realWorkbookDocuments and windows inside the running app; key events are created withNSEvent.keyEventand delivered toGridView.keyDown, mouse selection uses the grid's own geometry, saving and reopening goes through the document layer, printing is rendered to a PDF and its page count checked. - Menu and shortcut consistency:
testMenuBarHasNoDuplicateShortcutswalks the main menu and fails on any key equivalent used twice;-ORIXDumpMenus <json>writes the menu tree for the localization checks (localization.md). - Localization:
Scripts/localize.py --checkand theLocalizationTestssuites ofSheetUIandFeatureAudit.
What is done by eye
- Screenshots from the app itself. The app (
-ORIXShot) and every preview executable (--shot) render their window to a PNG withcacheDisplay(in:to:)and exit. These images were reviewed by a person in light and dark, English and French (accessibility.md lists which screens). Nothing compares them automatically: there is no snapshot or golden-image test.
Limits
- There is no XCUITest target. No test launches the app as a separate process and drives it through
the accessibility interface; Apple's
performAccessibilityAudit()was not run (accessibility.md). - Synthesized events are not real input. The app-hosted tests call
keyDownon the grid with constructedNSEvents and feed the rest of the text through the session "like the field editor does"; the system text-input path (input methods, dead keys, the real first-responder chain) is not exercised. - VoiceOver was not listened to; Full Keyboard Access, Voice Control and Switch Control were not exercised; Increase Contrast, Differentiate Without Colour and Reduce Motion were not switched on during a run.
- Settings, Page Setup, Help and the compatibility report were not put through the accessibility tree audit.
- No test opens a file written by ORIX Sheet in Microsoft Excel.
- Drag and drop, the open and save panels, printing to a physical printer, window restoration and multi-window behaviour have no automated test.
- Timing assertions are loose on purpose.
testLargeSheetStaysResponsiveasserts a worst repaint under 50 ms ("generous bound for CI noise") andtestMouseSelectionAndRedrawunder 100 ms; they catch regressions of an order of magnitude, not of a few percent. Use the benchmarks of performance.md for measurements.
7. Templates and automation
SheetTemplates, SheetAutomation and FeatureAutomation are built into the app (project.yml,
App/Sources/Features.swift) and are part of the Scripts/test-all.sh package list.
| Package | Test files | What they cover, by suite name |
|---|---|---|
| SheetTemplates | StructureTests.swift (Library, Structure), NumericTests.swift (Numeric), IOTests.swift (InputOutput), LibreOfficeTests.swift (LibreOfficeCrossCheck), ThumbnailTests.swift (Thumbnails), GeneratorInputs.swift (GeneratorInputs) |
structure of each template, key outputs against expected.json computed with numpy / scipy, save and reload, LibreOffice recalculation with one recorded accepted difference, thumbnail rendering. See ../templates.md |
| SheetAutomation | ModelTests.swift, EngineTests.swift, StepTests.swift, OutputTests.swift, TemplateMergeTests.swift (TemplateTests, MergeTests), ValidatorStoreTests.swift (ValidatorTests, StoreTests), CLITests.swift |
workflow model, engine, steps, outputs, template merge, validation, the on-disk store, the orix-sheet command line. See ../architecture/automation.md |
| FeatureAutomation | ModelTests.swift, EditingTests.swift (EditingTests, ScheduleTests), LocalizationIntentTests.swift (LocalizationTests, IntentServiceTests) |
the panel's view model, workflow editing, the LaunchAgent property list of a schedule, localization completeness, the service behind the App Intents |
Source: ls and grep -n '@Suite' over the three Tests directories.
8. Recorded test run
Copied from docs/reports/tests.json (run of 2026-11-21T08:47:01Z by Scripts/test-all.sh) and
docs/reports/app-tests.json (2026-11-21T08:48:35Z).
| Package | Test functions | Status | Seconds | Test targets |
|---|---|---|---|---|
| SheetCore | 32 | passed | 2 | SheetCoreTests |
| SheetFormat | 28 | passed | 4 | SheetFormatTests |
| FunctionsNumeric | 31 | passed | 7 | FunctionsNumericTests |
| FunctionsText | 19 | passed | 5 | FunctionsTextTests |
| FunctionsLookup | 49 | passed | 8 | FunctionsLookupTests |
| SheetEngine | 4 | passed | 5 | SheetEngineTests |
| SheetIO | 66 | passed | 4 | SheetIOTests |
| SheetCharts | 43 | passed | 10 | SheetChartsTests |
| SheetData | 64 | passed | 4 | SheetDataTests |
| SheetAnalytics | 310 | passed | 9 | StatisticsTests, FinanceModelsTests, WhatIfTests, AuditTests |
| SheetPython | 57 | passed | 12 | SheetPythonTests |
| SheetIntelligence | 62 | passed | 3 | SheetIntelligenceTests |
| SheetAutomation | 78 | passed | 4 | SheetAutomationTests |
| SheetTemplates | 29 | passed | 9 | SheetTemplatesTests |
| SheetUI | 20 | passed | 6 | SheetUITests |
| FeatureAudit | 60 | passed | 11 | FeatureAuditTests |
| FeatureCharts | 51 | passed | 10 | FeatureChartsTests |
| FeatureData | 57 | passed | 6 | FeatureDataTests |
| FeatureAnalysis | 125 | passed | 8 | AnalysisKitTests, StatsBasicTests, StatsModelsTests, FinanceStudioTests, WhatIfToolsTests |
| FeaturePython | 40 | passed | 16 | FeaturePythonTests |
| FeatureIntelligence | 17 | passed | 10 | FeatureIntelligenceTests |
| FeatureAutomation | 28 | passed | 6 | FeatureAutomationTests |
| Total, 22 packages | 1,270 | 0 failed packages | ||
| App (XCTest hosted in the app) | 9 | 0 failures | ORIXSheetTests |
The counts are test functions, as reported by the test runner (Test run with N tests). Many of
them are parameterised over fixtures: a function that replays a fixture file with thousands of
reference cases counts once. The number of reference cases behind the function libraries is in
../mathematics/reference-tests.md. A test disabled by a missing
tool or an unset environment variable (section 5) is still counted by the runner as a test of the run;
tests.json does not record how many were skipped.
9. Writing a new test
- Put pure logic in the library target and test it without AppKit. A view-model test that needs the main
actor is marked
@MainActor; suites that share global state (interpreters, the application-support directory, the accessibility API) are marked.serialized. - A test that needs an external tool must be disabled with a comment when the tool is missing
(
.enabled(if:)), never silently pass by returning early without saying so.RealModelTestsuses the.enabledform for its opt-in tests;LibreOfficeCrossCheckreturns early instead, and when touching it, prefer the.enabledform so that the skip is visible in the report. - A test that takes more than a few seconds belongs behind an environment variable (section 5) so that
Scripts/test-all.shstays usable as a pre-commit check. - Performance numbers are printed (
[bench],BENCHMARK,[timing]prefixes), not asserted, except for the loose bounds mentioned in section 6. - Never write expected values by running the code under test and pasting its output.