SPB Git

Repository Public

testing.md 24.2 KB

# Testing

The test strategy by layer, how to run the suites, where the fixtures come from, which tests are gated, and what the automated tests do not cover.

Related documents: setup.md, performance.md, accessibility.md, localization.md, ../mathematics/reference-tests.md (the reference sources and their case counts), ../release/checklist.md.

# 1. Where the numbers are

Test counts and pass/fail status change with every commit. The authoritative records are docs/reports/tests.json, written by Scripts/test-all.sh (see section 3), and docs/reports/app-tests.json for the app-hosted bundle. The table at the end of this document (section 8) is copied from those two files: 22 packages, 1,270 test functions, no failed package, plus 9 app-hosted tests with no failure.

# 2. Strategy by layer

flowchart TD
    A["App integration tests<br/>XCTest hosted in the app<br/>App/Tests"] --> B["Feature packages<br/>view models and adapters<br/>Feature*/Tests"]
    B --> C["UI kit<br/>session, layout, accessibility tree, contrast<br/>SheetUI/Tests"]
    B --> D["Domain libraries<br/>SheetAnalytics, SheetData, SheetCharts,<br/>SheetPython, SheetIntelligence, SheetIO"]
    B --> F["Function libraries<br/>FunctionsNumeric, FunctionsText, FunctionsLookup"]
    C --> E["Core<br/>SheetCore, SheetFormat, SheetEngine"]
    D --> E
    F --> E

SVG fallback: Test layers

Each box is tested by its own suite; an arrow means "is built on". The domain and function libraries are the two layers whose expected values come from external reference programs (see ../mathematics/reference-tests.md).

Layer Packages What the tests are External reference
Core model and engine SheetCore, SheetEngine hand-written expectations: addresses, parser, semantics, recalculation, editing with undo, input parsing, a performance smoke test; cross-library model and registry checks none
Number formats SheetFormat LibreOffice fixture replay with named deviation rules, hand-written edge cases, a fuzz test, a micro-benchmark suite LibreOffice
Worksheet functions FunctionsNumeric, FunctionsText, FunctionsLookup fixture replay (one parameterised test per fixture file), Microsoft documentation examples, hand-written edge cases, a test that fails when a registered function has no reference LibreOffice, SciPy, mpmath, NumPy, statsmodels, Python decimal, Excel-computed public data
File formats SheetIO read and write tests per format, ZIP and XML hardening (hostile archives, fuzzing of truncated and corrupted files), native-format integrity, interoperability with other programs, gated performance tests openpyxl, pandas, LibreOffice
Statistics, finance, what-if, audit SheetAnalytics (targets StatisticsTests, FinanceModelsTests, WhatIfTests, AuditTests, shared TestSupport) fixture replay against Python libraries; exact hand-derived expectations for the deterministic analyses SciPy, NumPy, pandas, statsmodels, scikit-learn, arch, Python decimal
Data engine SheetData engine, import, transform, pivot and sheet-bridge tests on seeded synthetic files pandas (no DuckDB on the reference side)
Charts SheetCharts numeric algorithms against Python references; every chart kind × difficult data rendered to SVG (parsed back with XMLParser), PDF and PNG; structural checks on the display list; vector-PDF heuristics; JSON round trips NumPy, SciPy, matplotlib
Python runtime SheetPython execution, API, discovery, records and security tests that run real interpreters; each sandbox denial is asserted, and the same probe is asserted to succeed without the sandbox the interpreters found on the Mac
Intelligence SheetIntelligence context building, validators and post-checks with a fake backend; RealModelTests against the on-device model, opt-in (ORIX_AI_EVAL=1, section 5) the on-device model (measurement, not a reference)
UI kit SheetUI AxisLayoutTests, SessionTests (edit / commit / undo, navigation, merge-aware selection, pointing, find, background recalculation, print pagination), ContrastTests, LocalizationTests, AccessibilityTreeTests WCAG 2.1 contrast formula
Feature panels FeatureAudit, FeatureCharts, FeatureData, FeatureAnalysis, FeaturePython, FeatureIntelligence, FeatureAutomation view models, providers and adapters driven without a window; localization completeness; integration with a real WorkbookSession —
Application App/Tests XCTest bundle hosted inside the running app: real documents, windows and grid, synthesized key and mouse events —
Templates SheetTemplates structure of each template, key outputs against expected.json, save and reload, LibreOffice recalculation of every template, thumbnails (section 7) numpy / scipy, LibreOffice
Automation engine SheetAutomation workflow model, engine, steps, outputs, template merge, validation, the on-disk store, the orix-sheet command line (section 7) —

Source: the test files under Packages/*/Tests and App/Tests (file and suite names), Packages/SheetCharts/README.md ("Tests"), Packages/SheetPython/SECURITY.md, Packages/SheetAnalytics/VALIDATION.md, Packages/SheetIO/COMPATIBILITY.md.

The packages use Swift Testing (@Test, @Suite, #expect); only the app-hosted bundle uses XCTest (final class AppIntegrationTests: XCTestCase).

# 3. Running the tests

# Everything, with a report

sh
Scripts/test-all.sh                      # every package, sequentially
Scripts/test-all.sh SheetCore SheetIO    # only the packages named
Scripts/build.sh test                    # same as Scripts/test-all.sh

What Scripts/test-all.sh does, read from the script:

  1. For each package, in this order — SheetCore SheetFormat FunctionsNumeric FunctionsText FunctionsLookup SheetEngine SheetIO SheetCharts SheetData SheetAnalytics SheetPython SheetIntelligence SheetAutomation SheetTemplates SheetUI FeatureAudit FeatureCharts FeatureData FeatureAnalysis FeaturePython FeatureIntelligence FeatureAutomation — it runs ORIX_AI_EVAL=0 swift test (a debug build) and stores the output, stripped of colour codes, in build/test-logs/<Package>.log. A package without a Package.swift is skipped.
  2. The test count is the sum of every Test run with N tests line (Swift Testing; a package may have several test targets) plus the last Executed N tests line (XCTest). A parameterised Swift Testing function counts once, whatever the number of arguments; a fixture file with thousands of cases is therefore one test.
  3. A package is passed when every such line contains "passed" and the log contains neither error: nor signal code; otherwise it is FAILED. A failing package does not stop the others.
  4. It prints one line per package (name, tests, status, seconds) and a total, and writes docs/reports/tests.json: {"date", "total", "failedPackages", "packages": [{"package", "tests", "status", "seconds"}]}.
  5. The exit status is 1 when at least one package failed.

The header comment of the script says it writes tests.json "+ a table"; the table is printed on standard output only, it is not written to a file.

The on-device model evaluation is opt-in (ORIX_AI_EVAL=1, section 5); any other value, including the 0 the script sets, leaves it skipped. It takes minutes (mean 28.2 s per request over 20 requests in Packages/SheetIntelligence/EVALUATION.md).

# One package, one test

sh
cd Packages/FunctionsLookup && swift test
cd Packages/SheetCore && swift test --filter RecalcTests
cd Packages/SheetIO && swift test --filter HardeningTests

Use --scratch-path <dir> when another build of the same package is running.

# The app-hosted tests

project.yml declares the ORIXSheetTests bundle (sources in App/Tests, TEST_HOST = the Debug app) and adds it to the test action of the ORIXSheet scheme, which passes -ApplePersistenceIgnoreState YES and -ORIXTesting YES to the host app. No script in Scripts/ runs it: Scripts/build.sh test and Scripts/test-all.sh cover the packages only. The command is:

sh
Scripts/build.sh generate
xcodebuild test -project ORIXSheet.xcodeproj -scheme ORIXSheet -destination 'platform=macOS,arch=arm64'

The recorded result is in docs/reports/app-tests.json: 9 tests, 0 failures (2026-11-21T08:48:35Z). It is a separate file; the app-hosted tests are not part of docs/reports/tests.json.

AppIntegrationTests.swift contains nine tests: features registered; typing formulas in the grid and undo; mouse selection and redraw; native save and reopen is lossless; foreign files are imported as untitled and export is preflighted; corrupted and unsupported files fail cleanly; printing produces a paginated vector PDF; the menu bar has no duplicate shortcuts; a large sheet (50 000 rows × 4 columns, 100 000 formulas) recalculates in the background and repaints under a bound.

# 4. Fixtures and generators

Reference values are committed; swift test needs neither Python nor LibreOffice to replay them.

Package Fixtures (Tests/<Target>/Fixtures/) Generators (Tests/<Target>/Generators/)
FunctionsNumeric 24 JSON files (lo_*.json, scipy_*.json, numpy_descriptive.json, statsmodels_regression.json, decimal_rounding.json, cmath_complex.json), 1.5 MiB gen_libreoffice.py, gen_scipy.py, common.py, data.py, overrides.py, coverage.py
FunctionsText function_references.json, 0.8 MiB generate_function_fixtures.py, lo_recalc.py
FunctionsLookup excel_reference.json, financial.json, securities.json, lookup_lo.json, arrays.json, 9.6 MiB gen_lookup.py, gen_arrays.py, gen_financial.py, gen_securities.py, gen_excel_reference.py, lo_harness.py, pyref.py
SheetFormat libreoffice_formats.json, 0.2 MiB generate_format_fixtures.py, lo_recalc.py
SheetAnalytics 7 JSON files for StatisticsTests, finance.json for FinanceModelsTests, 1.3 MiB gen_core.py, gen_descriptive_inference.py, gen_regression.py, gen_timeseries.py, gen_multivariate.py, gen_resampling.py, gen_tukey.py, gen_finance.py, gen_common.py
SheetData 14 files: CSV dialects, Parquet, JSON Lines, SQLite, Arrow, expected.json, 1.9 MiB generate_fixtures.py
SheetCharts reference.json, colormaps.json, 0.2 MiB gen_fixtures.py, gen_colormaps.py
SheetIO 7 files: XLSX from openpyxl and LibreOffice, an ODS, a raw XLSX, a macro workbook, 0.1 MiB gen_openpyxl_corpus.py, gen_raw_xlsx.py, gen_libreoffice_corpus.sh, dump_xlsx.py
SheetTemplates expected.json gen_expected.py (reads inputs.json)

Source: find Packages -path '*Fixtures*' -type f and find Packages -path '*/Generators/*', sizes summed with stat, run on 2026-11-21.

The other packages have no fixture directory: their tests build their inputs in code (synthetic workbooks, temporary files).

How each corpus is built, what the references are and how to regenerate them is described in ../mathematics/reference-tests.md. Two rules apply to anyone adding a function or a format:

  • add reference cases to the generator, not expected values copied from the implementation;
  • when a reference program is known to deviate from Excel's documented behaviour, record the Excel value and the reference's value with a reason (the overrides.py, excel source tag and ExcelDeviations.swift mechanisms), so that the deviation stays visible.

# 5. Gated and conditional tests

# Opt-in by environment variable

Variable Package / test Effect
ORIX_AI_EVAL=1 SheetIntelligence, RealModelTests.oversizedPromptMapsToContextWindowExceeded and RealModelTests.fullEvaluation enables the two tests that generate text with the on-device model; fullEvaluation runs the 20 requests and rewrites Packages/SheetIntelligence/EVALUATION.md. With the variable unset or set to anything else (0 included) both are reported as skipped, the model generates nothing and the file is not rewritten
ORIX_BENCH=1 SheetIO, PerformanceTests enables the 1,000,000-cell and 5,000,000-cell runs: ORIX_BENCH=1 swift test -c release -Xswiftc -enable-testing --filter PerformanceTests
ORIX_DATA_LARGE_TESTS=1 SheetData, millionRowPipeline enables the 1,000,000-row end-to-end test: ORIX_DATA_LARGE_TESTS=1 swift test --filter millionRow
FN_DUMP=<dir> FunctionsNumeric, ReferenceFixtureTests writes every mismatch of each fixture to <dir>/<fixture>.txt (to review a regenerated corpus)
ORIX_TEMPLATES_SKIP_LO SheetTemplates, LibreOfficeCrossCheck skips the LibreOffice recalculation of the templates
ORIX_TEMPLATES_INPUTS=<file> SheetTemplates, exportGeneratorInputs writes the inputs read by gen_expected.py
ORIX_TEMPLATES_DUMP=<file> SheetTemplates, dumpNamedValues debug aid: the value of every defined name of every template
ORIX_TEMPLATES_THUMBS=<dir> SheetTemplates, Thumbnails also writes the thumbnail PNGs
ORIX_TEMPLATES_NO_FEATURES, ORIX_TEMPLATES_NO_PREVIEW SheetTemplates/Package.swift build the package without the sibling feature packages / without the preview harness
ORIX_SHEET_SUPPORT_DIR SheetPython, SheetAutomation points the application-support directory at a temporary location

Source: grep -rn 'ProcessInfo.processInfo.environment' Packages --include='*.swift' and the files it lists, run on 2026-11-21.

# Disabled when a tool or a permission is missing

These tests are skipped, not failed, when their precondition is not met. A green run on a machine without the tools has therefore verified less than a green run on the build machine.

Condition Tests affected
/opt/homebrew/bin/python3 is not executable (Tools.hasPython) SheetIO: three InteropTests tests and one CSVTests test (the checks that run openpyxl and pandas on files SheetIO wrote)
/opt/homebrew/bin/soffice is not executable (Tools.hasLibreOffice) SheetIO: one InteropTests test and one ODSTests test; SheetTemplates: the LibreOffice cross-check returns early
No Python ≥ 3.9 interpreter found (Interp.any) most of SheetPython (execution, security, discovery, API) and four FeaturePython integration tests
No interpreter with pandas + numpy + matplotlib (Interp.scientific) the DataFrame, plotting and large-data tests of SheetPython and FeaturePython
No interpreter without pandas (Interp.bare, normally /usr/bin/python3) one SheetPython API test
polars not installed one SheetPython API test
The on-device model is unavailable (with ORIX_AI_EVAL=1) the two opt-in RealModelTests return early after printing the reason; EVALUATION.md is not written
The test runner is not trusted for the accessibility API (AXIsProcessTrusted()) the whole AccessibilityTreeTests suite of SheetUI

Source: grep -rn '\.enabled(' Packages/*/Tests, Packages/SheetIO/Tests/SheetIOTests/TestSupport.swift, Packages/SheetPython/Tests/SheetPythonTests/Support.swift, Packages/SheetIntelligence/Tests/SheetIntelligenceTests/RealModelTests.swift, Packages/SheetUI/Tests/SheetUITests/AccessibilityTests.swift.

Note on RealModelTests: availabilityIsReported always runs. It asks the system whether the model is available and generates nothing; that query is the only contact with the real backend in a plain swift test or a Scripts/test-all.sh run.

# 6. UI and automation testing

# What is automated

  • Session-level tests without a window (SheetUI/SessionTests): the WorkbookSession is the main actor object behind every window; editing, committing, undo, navigation, selection across merged cells, formula pointing, the find model, background recalculation and print pagination are tested on it directly.
  • Accessibility tree of a real view (SheetUI/AccessibilityTreeTests): a WorkbookView is put in an offscreen window and read through the system accessibility API; the numbers found (elements, controls, controls without a label) are reported in accessibility.md.
  • Contrast (SheetUI/ContrastTests): every text / background pair of the design tokens is measured in light and dark and the test fails under 4.5:1 for text and 3:1 for glyphs and structural lines.
  • Feature view models (Feature*/Tests): each panel exposes its state as a model that the tests drive without rendering; FeatureData adds a DataAutomation entry point used by its tests and by the data-preview benchmark.
  • App-hosted integration (App/Tests): real WorkbookDocuments and windows inside the running app; key events are created with NSEvent.keyEvent and delivered to GridView.keyDown, mouse selection uses the grid's own geometry, saving and reopening goes through the document layer, printing is rendered to a PDF and its page count checked.
  • Menu and shortcut consistency: testMenuBarHasNoDuplicateShortcuts walks the main menu and fails on any key equivalent used twice; -ORIXDumpMenus <json> writes the menu tree for the localization checks (localization.md).
  • Localization: Scripts/localize.py --check and the LocalizationTests suites of SheetUI and FeatureAudit.

# What is done by eye

  • Screenshots from the app itself. The app (-ORIXShot) and every preview executable (--shot) render their window to a PNG with cacheDisplay(in:to:) and exit. These images were reviewed by a person in light and dark, English and French (accessibility.md lists which screens). Nothing compares them automatically: there is no snapshot or golden-image test.

# Limits

  • There is no XCUITest target. No test launches the app as a separate process and drives it through the accessibility interface; Apple's performAccessibilityAudit() was not run (accessibility.md).
  • Synthesized events are not real input. The app-hosted tests call keyDown on the grid with constructed NSEvents and feed the rest of the text through the session "like the field editor does"; the system text-input path (input methods, dead keys, the real first-responder chain) is not exercised.
  • VoiceOver was not listened to; Full Keyboard Access, Voice Control and Switch Control were not exercised; Increase Contrast, Differentiate Without Colour and Reduce Motion were not switched on during a run.
  • Settings, Page Setup, Help and the compatibility report were not put through the accessibility tree audit.
  • No test opens a file written by ORIX Sheet in Microsoft Excel.
  • Drag and drop, the open and save panels, printing to a physical printer, window restoration and multi-window behaviour have no automated test.
  • Timing assertions are loose on purpose. testLargeSheetStaysResponsive asserts a worst repaint under 50 ms ("generous bound for CI noise") and testMouseSelectionAndRedraw under 100 ms; they catch regressions of an order of magnitude, not of a few percent. Use the benchmarks of performance.md for measurements.

# 7. Templates and automation

SheetTemplates, SheetAutomation and FeatureAutomation are built into the app (project.yml, App/Sources/Features.swift) and are part of the Scripts/test-all.sh package list.

Package Test files What they cover, by suite name
SheetTemplates StructureTests.swift (Library, Structure), NumericTests.swift (Numeric), IOTests.swift (InputOutput), LibreOfficeTests.swift (LibreOfficeCrossCheck), ThumbnailTests.swift (Thumbnails), GeneratorInputs.swift (GeneratorInputs) structure of each template, key outputs against expected.json computed with numpy / scipy, save and reload, LibreOffice recalculation with one recorded accepted difference, thumbnail rendering. See ../templates.md
SheetAutomation ModelTests.swift, EngineTests.swift, StepTests.swift, OutputTests.swift, TemplateMergeTests.swift (TemplateTests, MergeTests), ValidatorStoreTests.swift (ValidatorTests, StoreTests), CLITests.swift workflow model, engine, steps, outputs, template merge, validation, the on-disk store, the orix-sheet command line. See ../architecture/automation.md
FeatureAutomation ModelTests.swift, EditingTests.swift (EditingTests, ScheduleTests), LocalizationIntentTests.swift (LocalizationTests, IntentServiceTests) the panel's view model, workflow editing, the LaunchAgent property list of a schedule, localization completeness, the service behind the App Intents

Source: ls and grep -n '@Suite' over the three Tests directories.

# 8. Recorded test run

Copied from docs/reports/tests.json (run of 2026-11-21T08:47:01Z by Scripts/test-all.sh) and docs/reports/app-tests.json (2026-11-21T08:48:35Z).

Package Test functions Status Seconds Test targets
SheetCore 32 passed 2 SheetCoreTests
SheetFormat 28 passed 4 SheetFormatTests
FunctionsNumeric 31 passed 7 FunctionsNumericTests
FunctionsText 19 passed 5 FunctionsTextTests
FunctionsLookup 49 passed 8 FunctionsLookupTests
SheetEngine 4 passed 5 SheetEngineTests
SheetIO 66 passed 4 SheetIOTests
SheetCharts 43 passed 10 SheetChartsTests
SheetData 64 passed 4 SheetDataTests
SheetAnalytics 310 passed 9 StatisticsTests, FinanceModelsTests, WhatIfTests, AuditTests
SheetPython 57 passed 12 SheetPythonTests
SheetIntelligence 62 passed 3 SheetIntelligenceTests
SheetAutomation 78 passed 4 SheetAutomationTests
SheetTemplates 29 passed 9 SheetTemplatesTests
SheetUI 20 passed 6 SheetUITests
FeatureAudit 60 passed 11 FeatureAuditTests
FeatureCharts 51 passed 10 FeatureChartsTests
FeatureData 57 passed 6 FeatureDataTests
FeatureAnalysis 125 passed 8 AnalysisKitTests, StatsBasicTests, StatsModelsTests, FinanceStudioTests, WhatIfToolsTests
FeaturePython 40 passed 16 FeaturePythonTests
FeatureIntelligence 17 passed 10 FeatureIntelligenceTests
FeatureAutomation 28 passed 6 FeatureAutomationTests
Total, 22 packages 1,270 0 failed packages
App (XCTest hosted in the app) 9 0 failures ORIXSheetTests

The counts are test functions, as reported by the test runner (Test run with N tests). Many of them are parameterised over fixtures: a function that replays a fixture file with thousands of reference cases counts once. The number of reference cases behind the function libraries is in ../mathematics/reference-tests.md. A test disabled by a missing tool or an unset environment variable (section 5) is still counted by the runner as a test of the run; tests.json does not record how many were skipped.

# 9. Writing a new test

  • Put pure logic in the library target and test it without AppKit. A view-model test that needs the main actor is marked @MainActor; suites that share global state (interpreters, the application-support directory, the accessibility API) are marked .serialized.
  • A test that needs an external tool must be disabled with a comment when the tool is missing (.enabled(if:)), never silently pass by returning early without saying so. RealModelTests uses the .enabled form for its opt-in tests; LibreOfficeCrossCheck returns early instead, and when touching it, prefer the .enabled form so that the skip is visible in the report.
  • A test that takes more than a few seconds belongs behind an environment variable (section 5) so that Scripts/test-all.sh stays usable as a pre-commit check.
  • Performance numbers are printed ([bench], BENCHMARK, [timing] prefixes), not asserted, except for the loose bounds mentioned in section 6.
  • Never write expected values by running the code under test and pasting its output.
Files main
Menu

Theme