Testing
Calculation logic never imports SwiftUI, so every engine is tested from the command line without launching the app. The
UI layers are tested on the simulator, and the assembled app end to end with XCUITest. The mathematical references behind
the engine tests are described in docs/mathematics/reference-tests.md; project
setup and the dedicated simulators are in setup.md.
What runs where
| Layer | Runner | Platform | Framework |
|---|---|---|---|
Engine packages (ORIXNumerics, ORIXExpression, ORIXWorkspace, ORIXFinancial, ORIXStatistics, ORIXUnits, ORIXSymbolic, ORIXGraph, ORIXMathCamera) |
swift test in the package directory |
macOS | Swift Testing (@Suite, @Test, #expect) |
Feature packages (ORIXFeature*) |
xcodebuild test -scheme <package> |
iOS Simulator "ORIXCalc iPhone" | Swift Testing |
App unit tests (Tests/, target ORIXCalcTests) |
xcodebuild test -scheme ORIXCalc |
iOS Simulator, hosted in the app | Swift Testing |
UI tests (UITests/, target ORIXCalcUITests) |
same scheme | iOS Simulator | XCTest / XCUITest |
ORIXUI has no test target; its AppModel and store are exercised through the feature-package and app tests.
Run everything
Scripts/test-all.sh # engines, then features, then the app (unit + UI)
Scripts/test-all.sh engines # or: features | appThe script writes one log per target to build/logs/test-<name>.log and one summary line per target to
build/test-summary.txt (the last "Test run with …" line of Swift Testing, or "Executed N tests" for XCUITest). It ends
with RESULT: all suites passed or RESULT: FAILURES — see build/logs and exits non-zero on a failure. The app step
regenerates the Xcode project first. The results recorded for a release are kept in RELEASE-REPORT.md at the repository
root.
Run one package
# an engine
cd Packages/ORIXFinancial && swift test
# a feature package
cd Packages/ORIXFeatureGraph && xcodebuild test -scheme ORIXFeatureGraph \
-destination "platform=iOS Simulator,name=ORIXCalc iPhone" -derivedDataPath .build/xc
# the app (unit + UI tests)
Scripts/generate.sh && xcodebuild test -project ORIXCalc.xcodeproj -scheme ORIXCalc \
-destination "platform=iOS Simulator,name=ORIXCalc iPhone" -derivedDataPath build/DerivedDataUseful variants: swift test --filter <SuiteOrTestName>; ORIX_ACCURACY_REPORT=1 swift test in ORIXStatistics prints
the measured accuracy tables (SPC|… and ACC|… lines); swift test -c release -Xswiftc -enable-testing in ORIXGraph
runs the suite, including its timing assertions, against an optimised build.
Do not run two builds of the same package or of the app at the same time: they share .build/ and build/DerivedData.
Test targets
@Test counts are grep -rE '@Test\b' over each Tests/ directory; a parameterised or table-driven test counts once.
Engine packages
| Target | Location | @Test |
Covers |
|---|---|---|---|
ORIXNumericsTests |
Packages/ORIXNumerics/Tests/ |
12 | Root finding (RootFindingTests), quadrature and differentiation (CalculusTests) |
ORIXExpressionTests |
Packages/ORIXExpression/Tests/ |
37 | StandardArithmetic, Scientific, ComplexNumbers, Matrices, Formatting, Verification, RPN, Compiled, Regression |
ORIXWorkspaceTests |
Packages/ORIXWorkspace/Tests/ |
11 | Reactive recalculation (Reactive), exports (Exports), the .orixcalc file (NativeFile) |
ORIXFinancialTests |
Packages/ORIXFinancial/Tests/ |
67 | TVMTests, FinancialDateTests, CashFlowTests, LoanTests, BondTests, InvestmentTests, OptionTests, SensitivityTests; reference file Resources/reference.json |
ORIXStatisticsTests |
Packages/ORIXStatistics/Tests/ |
85 | Special functions and distributions against the mpmath grids, descriptive statistics, inference, regression, random generators and samplers, Monte Carlo, scenarios, bootstrap; reference files in ReferenceData/ |
ORIXUnitsTests |
Packages/ORIXUnits/Tests/ |
60 | Catalog integrity, exact arithmetic, exactness, temperature, fuel economy, data prefixes, the Double path, lookup and search, rate conventions, currency and the ECB parser, the Python reference table |
ORIXSymbolicTests |
Packages/ORIXSymbolic/Tests/ |
58 | ReferenceComparison (sympy / mpmath), CanonicalForms, Calculus, Solving, PolynomialAlgebra, Systems, Optimisation, DifferentialEquations, RandomisedSimplification, BoundsAndPerformance |
ORIXGraphTests |
Packages/ORIXGraph/Tests/ |
117 | Explicit sampler (PolesAndJumps, DomainsAndHoles, Oscillation, Accuracy, ExtremeRanges, AwkwardFunctions), ParametricAndPolar, ImplicitCurves, Surfaces, Camera, analysis (RootsAndIntersections, Extrema, Derivatives, Integrals, TraceAndTable), ViewportTests, Ticks, ParserTests, DataAndExport, Performance |
ORIXMathCameraTests |
Packages/ORIXMathCamera/Tests/ |
3 | Text normalisation and confusion repair (NormalizerTests), the recognition benchmark (RecognitionBenchmark) |
Feature packages
| Target | Location | @Test |
Covers |
|---|---|---|---|
ORIXFeatureCalculatorTests |
Packages/ORIXFeatureCalculator/Tests/ |
11 | Expression editor (EditorTests), calculator model (CalculatorModelTests) |
ORIXFeatureFinancialTests |
Packages/ORIXFeatureFinancial/Tests/ |
35 | TVM keypad, cash-flow model, loan model, bonds / investment / options, sensitivity and saved models |
ORIXFeatureGraphTests |
Packages/ORIXFeatureGraph/Tests/ |
71 | Entry parsing, sliders, viewport, sessions, 3D model, analysis and previews, pixel-level render tests, layout screenshots |
ORIXFeatureMathTests |
Packages/ORIXFeatureMath/Tests/ |
48 | Tool answers, angle modes, store, keyboard, math layout tree and off-screen formula renders |
ORIXFeatureStatisticsTests |
Packages/ORIXFeatureStatistics/Tests/ |
66 | Paste parser and data model, distributions and native functions, inference, regression, simulation, saved state, layout renders |
ORIXFeatureConvertTests |
Packages/ORIXFeatureConvert/Tests/ |
46 | Keypad entry, exact conversion, selection, hand-off and history, temperature and fuel, rate conventions, currency model (ECB fixture), render smoke test |
ORIXFeatureWorkspaceTests |
Packages/ORIXFeatureWorkspace/Tests/ |
46 | Sheet reactivity, structure, undo and editing, library, auto-save, hand-off, import, templates, exports, layout renders |
App
| Target | Location | Tests | Covers |
|---|---|---|---|
ORIXCalcTests |
Tests/AppTests.swift |
5 @Test |
Launch options, shared history and variables, backup round trip, opening a workspace file, settings driving formatting |
ORIXCalcUITests |
UITests/ORIXCalcUITests.swift |
6 XCTest methods | Arithmetic on the keypad, division by zero shown as an error, every environment opens, shared and reusable history, an exact angle in Scientific, the accessibility audit |
Reference data
Engine tests that compare against independent references load files generated by Python scripts kept next to the code. Regenerate after changing a convention or adding a case, then re-run the package's tests:
| Engine | Regenerate | Then |
|---|---|---|
| Financial | python3 Packages/ORIXFinancial/Reference/generate_reference.py |
The script copies reference.json into Tests/ORIXFinancialTests/Resources/; bundledReferenceMatchesTheGeneratedFile fails if the two copies differ |
| Statistics | Packages/ORIXStatistics/Reference/generate_all.sh |
Runs the six gen_*.py scripts (seeded), writes Reference/output/ with VERSIONS.txt, mirrored into Tests/ORIXStatisticsTests/ReferenceData/ |
| Units | python3 Packages/ORIXUnits/Reference/generate_reference.py |
Writes the two .tsv tables; the same table is embedded in Tests/ORIXUnitsTests/ReferenceData.swift, and a coverage test fails if the catalog and the reference do not list the same unit ids |
| Symbolic | python3 Packages/ORIXSymbolic/Reference/generate_reference.py |
Writes Reference/reference.json, copied into Tests/ORIXSymbolicTests/Resources/ |
The scripts need mpmath, numpy, scipy, statsmodels and sympy as applicable; the versions used are recorded in each
VALIDATION.md. Reference files are committed, so running the tests does not require Python.
Render and pixel tests
Feature packages render their real views off-screen on the simulator.
- Graph — pixel proof (
Packages/ORIXFeatureGraph/Tests/ORIXFeatureGraphTests/RenderTests.swift, helpers inSupport.swift). The sharedGraphRendereris rasterised withImageRendererinto an RGBA bitmap and the pixels are inspected: the sine is drawn along its path;1/xhas no stroke in the column of its asymptote;tan xhas none at any asymptote;floor(x)has no risers (a riser would be a 40-pixel vertical run in an integer column); asymptote guides are lighter than the curve; an aliased function is a band with a notice; hole markers, inequality fills and shaded regions render; dark appearance uses the dark tokens.ScreenshotTestsrenders the screen layouts. - Math (
MathNotationTests.swift): the layout tree of fractions, powers, radicals and matrices, spoken descriptions, measured geometry, and off-screen renders of formulas counted by inked pixels. - Statistics, Convert, Workspace (
RenderTests.swift,RenderSmokeTests.swift): every section or layout is rendered at phone and tablet sizes in light, dark and accessibility text sizes as a smoke test; the images are written to the temporary directory (Workspace: PNG files) for visual review.
Accessibility audit
testAccessibilityAuditOfTheCalculator in UITests/ORIXCalcUITests.swift launches the Standard calculator and runs
XCUIApplication.performAccessibilityAudit(for: [.dynamicType, .sufficientElementDescription, .hitRegion]). The issue
handler ignores only the decorative separators inside the display, which are not interactive elements.
Math Camera recognition benchmark
RecognitionBenchmark.printedLinearExpressions in
Packages/ORIXMathCamera/Tests/ORIXMathCameraTests/RecognitionTests.swift renders 30 printed expressions in three fonts
(Helvetica, Times New Roman, Menlo) with Core Text, runs MathRecognizer.recognize on each image and counts a hit when
the top candidate parses and evaluates to the same value as the source expression. It prints
MATH-CAMERA-BENCHMARK total=… parsed=… correct=… rate=…
MATH-CAMERA-MISS <font>: <expression> → <raw> ⇒ <normalized>and asserts a floor of rate >= 0.80, which guards against regressions; the feature ships as a reviewed-entry aid, not as
an automatic solver. Read the measured rate from the test log (build/logs/test-ORIXMathCamera.log).
Performance
- Engine benchmark.
cd Benchmarks && swift run -c release orix-benchruns the expression, workspace and domain benchmarks (financial, statistics, graph, units) single-threaded and prints a Markdown table with the machine and build configuration. The recorded output isbenchmark-results.md. - Graph timings.
Packages/ORIXGraph/Tests/ORIXGraphTests/PerformanceTests.swiftprintsPERF|…andCOUNT|…lines and asserts time limits that are generous in debug builds and tight in release builds. - Symbolic bounds.
BoundsAndPerformanceasserts size limits and a simplification time bound.
Regression tests
Rule: every bug fix adds a regression test. The existing ones sit in the test target of the package that had the bug and carry a comment saying where the bug was reported.
Existing examples:
| Test | Location | Bug it pins |
|---|---|---|
Regression.equationWithBuiltinOnTheLeft |
Packages/ORIXExpression/Tests/ORIXExpressionTests/EngineTests.swift |
Reported by the graph engine: an equation whose left side is a built-in call (sin(x) = cos(y)) was read as a function definition |
Regression.longChainsOnASmallStack |
same | Reported by the symbolic engine: a sum of 900 terms overflowed the 512 KB stack of a secondary thread; also asserts that a 6000-term chain is refused with an error |
acklamCoefficientsAreTheRealOnes |
Packages/ORIXStatistics/Tests/ORIXStatisticsTests/SpecialFunctionTests.swift |
Guards against a mistyped coefficient: the raw approximation must show its published 1.15e-9 bound |
bundledReferenceMatchesTheGeneratedFile |
Packages/ORIXFinancial/Tests/ORIXFinancialTests/ |
Guards against a stale bundled reference file |
RecognitionBenchmark floor |
Packages/ORIXMathCamera/Tests/ORIXMathCameraTests/RecognitionTests.swift |
Guards the recognition rate |
Conventions
- Tests state the tolerance they assert; the asserted tolerances are listed per engine in each
VALIDATION.md. - Tests that touch
AppModeluse an in-memory store and a uniqueUserDefaultssuite, so they never read or write the user's data. The launch helper of the UI tests passes-ORIXInMemory 1. - Engine tests do no networking; the ECB parser is tested against a bundled fixture.
- Error paths are tested as errors (
#expect(throws:)), not as special values.