SPB Git

Repository Public MIT

testing.md 13.1 KB

# Testing

Calculation logic never imports SwiftUI, so every engine is tested from the command line without launching the app. The UI layers are tested on the simulator, and the assembled app end to end with XCUITest. The mathematical references behind the engine tests are described in docs/mathematics/reference-tests.md; project setup and the dedicated simulators are in setup.md.

# What runs where

Layer Runner Platform Framework
Engine packages (ORIXNumerics, ORIXExpression, ORIXWorkspace, ORIXFinancial, ORIXStatistics, ORIXUnits, ORIXSymbolic, ORIXGraph, ORIXMathCamera) swift test in the package directory macOS Swift Testing (@Suite, @Test, #expect)
Feature packages (ORIXFeature*) xcodebuild test -scheme <package> iOS Simulator "ORIXCalc iPhone" Swift Testing
App unit tests (Tests/, target ORIXCalcTests) xcodebuild test -scheme ORIXCalc iOS Simulator, hosted in the app Swift Testing
UI tests (UITests/, target ORIXCalcUITests) same scheme iOS Simulator XCTest / XCUITest

ORIXUI has no test target; its AppModel and store are exercised through the feature-package and app tests.

# Run everything

sh
Scripts/test-all.sh            # engines, then features, then the app (unit + UI)
Scripts/test-all.sh engines    # or: features | app

The script writes one log per target to build/logs/test-<name>.log and one summary line per target to build/test-summary.txt (the last "Test run with …" line of Swift Testing, or "Executed N tests" for XCUITest). It ends with RESULT: all suites passed or RESULT: FAILURES — see build/logs and exits non-zero on a failure. The app step regenerates the Xcode project first. The results recorded for a release are kept in RELEASE-REPORT.md at the repository root.

# Run one package

sh
# an engine
cd Packages/ORIXFinancial && swift test

# a feature package
cd Packages/ORIXFeatureGraph && xcodebuild test -scheme ORIXFeatureGraph \
  -destination "platform=iOS Simulator,name=ORIXCalc iPhone" -derivedDataPath .build/xc

# the app (unit + UI tests)
Scripts/generate.sh && xcodebuild test -project ORIXCalc.xcodeproj -scheme ORIXCalc \
  -destination "platform=iOS Simulator,name=ORIXCalc iPhone" -derivedDataPath build/DerivedData

Useful variants: swift test --filter <SuiteOrTestName>; ORIX_ACCURACY_REPORT=1 swift test in ORIXStatistics prints the measured accuracy tables (SPC|… and ACC|… lines); swift test -c release -Xswiftc -enable-testing in ORIXGraph runs the suite, including its timing assertions, against an optimised build.

Do not run two builds of the same package or of the app at the same time: they share .build/ and build/DerivedData.

# Test targets

@Test counts are grep -rE '@Test\b' over each Tests/ directory; a parameterised or table-driven test counts once.

# Engine packages

Target Location @Test Covers
ORIXNumericsTests Packages/ORIXNumerics/Tests/ 12 Root finding (RootFindingTests), quadrature and differentiation (CalculusTests)
ORIXExpressionTests Packages/ORIXExpression/Tests/ 37 StandardArithmetic, Scientific, ComplexNumbers, Matrices, Formatting, Verification, RPN, Compiled, Regression
ORIXWorkspaceTests Packages/ORIXWorkspace/Tests/ 11 Reactive recalculation (Reactive), exports (Exports), the .orixcalc file (NativeFile)
ORIXFinancialTests Packages/ORIXFinancial/Tests/ 67 TVMTests, FinancialDateTests, CashFlowTests, LoanTests, BondTests, InvestmentTests, OptionTests, SensitivityTests; reference file Resources/reference.json
ORIXStatisticsTests Packages/ORIXStatistics/Tests/ 85 Special functions and distributions against the mpmath grids, descriptive statistics, inference, regression, random generators and samplers, Monte Carlo, scenarios, bootstrap; reference files in ReferenceData/
ORIXUnitsTests Packages/ORIXUnits/Tests/ 60 Catalog integrity, exact arithmetic, exactness, temperature, fuel economy, data prefixes, the Double path, lookup and search, rate conventions, currency and the ECB parser, the Python reference table
ORIXSymbolicTests Packages/ORIXSymbolic/Tests/ 58 ReferenceComparison (sympy / mpmath), CanonicalForms, Calculus, Solving, PolynomialAlgebra, Systems, Optimisation, DifferentialEquations, RandomisedSimplification, BoundsAndPerformance
ORIXGraphTests Packages/ORIXGraph/Tests/ 117 Explicit sampler (PolesAndJumps, DomainsAndHoles, Oscillation, Accuracy, ExtremeRanges, AwkwardFunctions), ParametricAndPolar, ImplicitCurves, Surfaces, Camera, analysis (RootsAndIntersections, Extrema, Derivatives, Integrals, TraceAndTable), ViewportTests, Ticks, ParserTests, DataAndExport, Performance
ORIXMathCameraTests Packages/ORIXMathCamera/Tests/ 3 Text normalisation and confusion repair (NormalizerTests), the recognition benchmark (RecognitionBenchmark)

# Feature packages

Target Location @Test Covers
ORIXFeatureCalculatorTests Packages/ORIXFeatureCalculator/Tests/ 11 Expression editor (EditorTests), calculator model (CalculatorModelTests)
ORIXFeatureFinancialTests Packages/ORIXFeatureFinancial/Tests/ 35 TVM keypad, cash-flow model, loan model, bonds / investment / options, sensitivity and saved models
ORIXFeatureGraphTests Packages/ORIXFeatureGraph/Tests/ 71 Entry parsing, sliders, viewport, sessions, 3D model, analysis and previews, pixel-level render tests, layout screenshots
ORIXFeatureMathTests Packages/ORIXFeatureMath/Tests/ 48 Tool answers, angle modes, store, keyboard, math layout tree and off-screen formula renders
ORIXFeatureStatisticsTests Packages/ORIXFeatureStatistics/Tests/ 66 Paste parser and data model, distributions and native functions, inference, regression, simulation, saved state, layout renders
ORIXFeatureConvertTests Packages/ORIXFeatureConvert/Tests/ 46 Keypad entry, exact conversion, selection, hand-off and history, temperature and fuel, rate conventions, currency model (ECB fixture), render smoke test
ORIXFeatureWorkspaceTests Packages/ORIXFeatureWorkspace/Tests/ 46 Sheet reactivity, structure, undo and editing, library, auto-save, hand-off, import, templates, exports, layout renders

# App

Target Location Tests Covers
ORIXCalcTests Tests/AppTests.swift 5 @Test Launch options, shared history and variables, backup round trip, opening a workspace file, settings driving formatting
ORIXCalcUITests UITests/ORIXCalcUITests.swift 6 XCTest methods Arithmetic on the keypad, division by zero shown as an error, every environment opens, shared and reusable history, an exact angle in Scientific, the accessibility audit

# Reference data

Engine tests that compare against independent references load files generated by Python scripts kept next to the code. Regenerate after changing a convention or adding a case, then re-run the package's tests:

Engine Regenerate Then
Financial python3 Packages/ORIXFinancial/Reference/generate_reference.py The script copies reference.json into Tests/ORIXFinancialTests/Resources/; bundledReferenceMatchesTheGeneratedFile fails if the two copies differ
Statistics Packages/ORIXStatistics/Reference/generate_all.sh Runs the six gen_*.py scripts (seeded), writes Reference/output/ with VERSIONS.txt, mirrored into Tests/ORIXStatisticsTests/ReferenceData/
Units python3 Packages/ORIXUnits/Reference/generate_reference.py Writes the two .tsv tables; the same table is embedded in Tests/ORIXUnitsTests/ReferenceData.swift, and a coverage test fails if the catalog and the reference do not list the same unit ids
Symbolic python3 Packages/ORIXSymbolic/Reference/generate_reference.py Writes Reference/reference.json, copied into Tests/ORIXSymbolicTests/Resources/

The scripts need mpmath, numpy, scipy, statsmodels and sympy as applicable; the versions used are recorded in each VALIDATION.md. Reference files are committed, so running the tests does not require Python.

# Render and pixel tests

Feature packages render their real views off-screen on the simulator.

  • Graph — pixel proof (Packages/ORIXFeatureGraph/Tests/ORIXFeatureGraphTests/RenderTests.swift, helpers in Support.swift). The shared GraphRenderer is rasterised with ImageRenderer into an RGBA bitmap and the pixels are inspected: the sine is drawn along its path; 1/x has no stroke in the column of its asymptote; tan x has none at any asymptote; floor(x) has no risers (a riser would be a 40-pixel vertical run in an integer column); asymptote guides are lighter than the curve; an aliased function is a band with a notice; hole markers, inequality fills and shaded regions render; dark appearance uses the dark tokens. ScreenshotTests renders the screen layouts.
  • Math (MathNotationTests.swift): the layout tree of fractions, powers, radicals and matrices, spoken descriptions, measured geometry, and off-screen renders of formulas counted by inked pixels.
  • Statistics, Convert, Workspace (RenderTests.swift, RenderSmokeTests.swift): every section or layout is rendered at phone and tablet sizes in light, dark and accessibility text sizes as a smoke test; the images are written to the temporary directory (Workspace: PNG files) for visual review.

# Accessibility audit

testAccessibilityAuditOfTheCalculator in UITests/ORIXCalcUITests.swift launches the Standard calculator and runs XCUIApplication.performAccessibilityAudit(for: [.dynamicType, .sufficientElementDescription, .hitRegion]). The issue handler ignores only the decorative separators inside the display, which are not interactive elements.

# Math Camera recognition benchmark

RecognitionBenchmark.printedLinearExpressions in Packages/ORIXMathCamera/Tests/ORIXMathCameraTests/RecognitionTests.swift renders 30 printed expressions in three fonts (Helvetica, Times New Roman, Menlo) with Core Text, runs MathRecognizer.recognize on each image and counts a hit when the top candidate parses and evaluates to the same value as the source expression. It prints

text
MATH-CAMERA-BENCHMARK total=… parsed=… correct=… rate=…
MATH-CAMERA-MISS <font>: <expression> → <raw> ⇒ <normalized>

and asserts a floor of rate >= 0.80, which guards against regressions; the feature ships as a reviewed-entry aid, not as an automatic solver. Read the measured rate from the test log (build/logs/test-ORIXMathCamera.log).

# Performance

  • Engine benchmark. cd Benchmarks && swift run -c release orix-bench runs the expression, workspace and domain benchmarks (financial, statistics, graph, units) single-threaded and prints a Markdown table with the machine and build configuration. The recorded output is benchmark-results.md.
  • Graph timings. Packages/ORIXGraph/Tests/ORIXGraphTests/PerformanceTests.swift prints PERF|… and COUNT|… lines and asserts time limits that are generous in debug builds and tight in release builds.
  • Symbolic bounds. BoundsAndPerformance asserts size limits and a simplification time bound.

# Regression tests

Rule: every bug fix adds a regression test. The existing ones sit in the test target of the package that had the bug and carry a comment saying where the bug was reported.

Existing examples:

Test Location Bug it pins
Regression.equationWithBuiltinOnTheLeft Packages/ORIXExpression/Tests/ORIXExpressionTests/EngineTests.swift Reported by the graph engine: an equation whose left side is a built-in call (sin(x) = cos(y)) was read as a function definition
Regression.longChainsOnASmallStack same Reported by the symbolic engine: a sum of 900 terms overflowed the 512 KB stack of a secondary thread; also asserts that a 6000-term chain is refused with an error
acklamCoefficientsAreTheRealOnes Packages/ORIXStatistics/Tests/ORIXStatisticsTests/SpecialFunctionTests.swift Guards against a mistyped coefficient: the raw approximation must show its published 1.15e-9 bound
bundledReferenceMatchesTheGeneratedFile Packages/ORIXFinancial/Tests/ORIXFinancialTests/ Guards against a stale bundled reference file
RecognitionBenchmark floor Packages/ORIXMathCamera/Tests/ORIXMathCameraTests/RecognitionTests.swift Guards the recognition rate

# Conventions

  • Tests state the tolerance they assert; the asserted tolerances are listed per engine in each VALIDATION.md.
  • Tests that touch AppModel use an in-memory store and a unique UserDefaults suite, so they never read or write the user's data. The launch helper of the UI tests passes -ORIXInMemory 1.
  • Engine tests do no networking; the ECB parser is tested against a bundled fixture.
  • Error paths are tested as errors (#expect(throws:)), not as special values.
Files main
Menu

Theme