Repository Public MIT
1<!--2ORIX Calc3File: docs/development/testing.md4Author: Simon-Pierre Boucher5Contact: contact@spboucher.ai6-->7# Testing89Calculation logic never imports SwiftUI, so every engine is tested from the command line without launching the app. The10UI layers are tested on the simulator, and the assembled app end to end with XCUITest. The mathematical references behind11the engine tests are described in [`docs/mathematics/reference-tests.md`](../mathematics/reference-tests.md); project12setup and the dedicated simulators are in [`setup.md`](setup.md).1314## What runs where1516| Layer | Runner | Platform | Framework |17|---|---|---|---|18| Engine packages (`ORIXNumerics`, `ORIXExpression`, `ORIXWorkspace`, `ORIXFinancial`, `ORIXStatistics`, `ORIXUnits`, `ORIXSymbolic`, `ORIXGraph`, `ORIXMathCamera`) | `swift test` in the package directory | macOS | Swift Testing (`@Suite`, `@Test`, `#expect`) |19| Feature packages (`ORIXFeature*`) | `xcodebuild test -scheme <package>` | iOS Simulator "ORIXCalc iPhone" | Swift Testing |20| App unit tests (`Tests/`, target `ORIXCalcTests`) | `xcodebuild test -scheme ORIXCalc` | iOS Simulator, hosted in the app | Swift Testing |21| UI tests (`UITests/`, target `ORIXCalcUITests`) | same scheme | iOS Simulator | XCTest / XCUITest |2223`ORIXUI` has no test target; its `AppModel` and store are exercised through the feature-package and app tests.2425## Run everything2627```sh28Scripts/test-all.sh # engines, then features, then the app (unit + UI)29Scripts/test-all.sh engines # or: features | app30```3132The script writes one log per target to `build/logs/test-<name>.log` and one summary line per target to33`build/test-summary.txt` (the last "Test run with …" line of Swift Testing, or "Executed N tests" for XCUITest). It ends34with `RESULT: all suites passed` or `RESULT: FAILURES — see build/logs` and exits non-zero on a failure. The `app` step35regenerates the Xcode project first. The results recorded for a release are kept in `RELEASE-REPORT.md` at the repository36root.3738## Run one package3940```sh41# an engine42cd Packages/ORIXFinancial && swift test4344# a feature package45cd Packages/ORIXFeatureGraph && xcodebuild test -scheme ORIXFeatureGraph \46 -destination "platform=iOS Simulator,name=ORIXCalc iPhone" -derivedDataPath .build/xc4748# the app (unit + UI tests)49Scripts/generate.sh && xcodebuild test -project ORIXCalc.xcodeproj -scheme ORIXCalc \50 -destination "platform=iOS Simulator,name=ORIXCalc iPhone" -derivedDataPath build/DerivedData51```5253Useful variants: `swift test --filter <SuiteOrTestName>`; `ORIX_ACCURACY_REPORT=1 swift test` in `ORIXStatistics` prints54the measured accuracy tables (`SPC|…` and `ACC|…` lines); `swift test -c release -Xswiftc -enable-testing` in `ORIXGraph`55runs the suite, including its timing assertions, against an optimised build.5657Do not run two builds of the same package or of the app at the same time: they share `.build/` and `build/DerivedData`.5859## Test targets6061`@Test` counts are `grep -rE '@Test\b'` over each `Tests/` directory; a parameterised or table-driven test counts once.6263### Engine packages6465| Target | Location | `@Test` | Covers |66|---|---|--:|---|67| `ORIXNumericsTests` | `Packages/ORIXNumerics/Tests/` | 12 | Root finding (`RootFindingTests`), quadrature and differentiation (`CalculusTests`) |68| `ORIXExpressionTests` | `Packages/ORIXExpression/Tests/` | 37 | `StandardArithmetic`, `Scientific`, `ComplexNumbers`, `Matrices`, `Formatting`, `Verification`, `RPN`, `Compiled`, `Regression` |69| `ORIXWorkspaceTests` | `Packages/ORIXWorkspace/Tests/` | 11 | Reactive recalculation (`Reactive`), exports (`Exports`), the `.orixcalc` file (`NativeFile`) |70| `ORIXFinancialTests` | `Packages/ORIXFinancial/Tests/` | 67 | `TVMTests`, `FinancialDateTests`, `CashFlowTests`, `LoanTests`, `BondTests`, `InvestmentTests`, `OptionTests`, `SensitivityTests`; reference file `Resources/reference.json` |71| `ORIXStatisticsTests` | `Packages/ORIXStatistics/Tests/` | 85 | Special functions and distributions against the mpmath grids, descriptive statistics, inference, regression, random generators and samplers, Monte Carlo, scenarios, bootstrap; reference files in `ReferenceData/` |72| `ORIXUnitsTests` | `Packages/ORIXUnits/Tests/` | 60 | Catalog integrity, exact arithmetic, exactness, temperature, fuel economy, data prefixes, the `Double` path, lookup and search, rate conventions, currency and the ECB parser, the Python reference table |73| `ORIXSymbolicTests` | `Packages/ORIXSymbolic/Tests/` | 58 | `ReferenceComparison` (sympy / mpmath), `CanonicalForms`, `Calculus`, `Solving`, `PolynomialAlgebra`, `Systems`, `Optimisation`, `DifferentialEquations`, `RandomisedSimplification`, `BoundsAndPerformance` |74| `ORIXGraphTests` | `Packages/ORIXGraph/Tests/` | 117 | Explicit sampler (`PolesAndJumps`, `DomainsAndHoles`, `Oscillation`, `Accuracy`, `ExtremeRanges`, `AwkwardFunctions`), `ParametricAndPolar`, `ImplicitCurves`, `Surfaces`, `Camera`, analysis (`RootsAndIntersections`, `Extrema`, `Derivatives`, `Integrals`, `TraceAndTable`), `ViewportTests`, `Ticks`, `ParserTests`, `DataAndExport`, `Performance` |75| `ORIXMathCameraTests` | `Packages/ORIXMathCamera/Tests/` | 3 | Text normalisation and confusion repair (`NormalizerTests`), the recognition benchmark (`RecognitionBenchmark`) |7677### Feature packages7879| Target | Location | `@Test` | Covers |80|---|---|--:|---|81| `ORIXFeatureCalculatorTests` | `Packages/ORIXFeatureCalculator/Tests/` | 11 | Expression editor (`EditorTests`), calculator model (`CalculatorModelTests`) |82| `ORIXFeatureFinancialTests` | `Packages/ORIXFeatureFinancial/Tests/` | 35 | TVM keypad, cash-flow model, loan model, bonds / investment / options, sensitivity and saved models |83| `ORIXFeatureGraphTests` | `Packages/ORIXFeatureGraph/Tests/` | 71 | Entry parsing, sliders, viewport, sessions, 3D model, analysis and previews, pixel-level render tests, layout screenshots |84| `ORIXFeatureMathTests` | `Packages/ORIXFeatureMath/Tests/` | 48 | Tool answers, angle modes, store, keyboard, math layout tree and off-screen formula renders |85| `ORIXFeatureStatisticsTests` | `Packages/ORIXFeatureStatistics/Tests/` | 66 | Paste parser and data model, distributions and native functions, inference, regression, simulation, saved state, layout renders |86| `ORIXFeatureConvertTests` | `Packages/ORIXFeatureConvert/Tests/` | 46 | Keypad entry, exact conversion, selection, hand-off and history, temperature and fuel, rate conventions, currency model (ECB fixture), render smoke test |87| `ORIXFeatureWorkspaceTests` | `Packages/ORIXFeatureWorkspace/Tests/` | 46 | Sheet reactivity, structure, undo and editing, library, auto-save, hand-off, import, templates, exports, layout renders |8889### App9091| Target | Location | Tests | Covers |92|---|---|--:|---|93| `ORIXCalcTests` | `Tests/AppTests.swift` | 5 `@Test` | Launch options, shared history and variables, backup round trip, opening a workspace file, settings driving formatting |94| `ORIXCalcUITests` | `UITests/ORIXCalcUITests.swift` | 6 XCTest methods | Arithmetic on the keypad, division by zero shown as an error, every environment opens, shared and reusable history, an exact angle in Scientific, the accessibility audit |9596## Reference data9798Engine tests that compare against independent references load files generated by Python scripts kept next to the code.99Regenerate after changing a convention or adding a case, then re-run the package's tests:100101| Engine | Regenerate | Then |102|---|---|---|103| Financial | `python3 Packages/ORIXFinancial/Reference/generate_reference.py` | The script copies `reference.json` into `Tests/ORIXFinancialTests/Resources/`; `bundledReferenceMatchesTheGeneratedFile` fails if the two copies differ |104| Statistics | `Packages/ORIXStatistics/Reference/generate_all.sh` | Runs the six `gen_*.py` scripts (seeded), writes `Reference/output/` with `VERSIONS.txt`, mirrored into `Tests/ORIXStatisticsTests/ReferenceData/` |105| Units | `python3 Packages/ORIXUnits/Reference/generate_reference.py` | Writes the two `.tsv` tables; the same table is embedded in `Tests/ORIXUnitsTests/ReferenceData.swift`, and a coverage test fails if the catalog and the reference do not list the same unit ids |106| Symbolic | `python3 Packages/ORIXSymbolic/Reference/generate_reference.py` | Writes `Reference/reference.json`, copied into `Tests/ORIXSymbolicTests/Resources/` |107108The scripts need mpmath, numpy, scipy, statsmodels and sympy as applicable; the versions used are recorded in each109`VALIDATION.md`. Reference files are committed, so running the tests does not require Python.110111## Render and pixel tests112113Feature packages render their real views off-screen on the simulator.114115- **Graph — pixel proof** (`Packages/ORIXFeatureGraph/Tests/ORIXFeatureGraphTests/RenderTests.swift`, helpers in116 `Support.swift`). The shared `GraphRenderer` is rasterised with `ImageRenderer` into an RGBA bitmap and the pixels are117 inspected: the sine is drawn along its path; `1/x` has no stroke in the column of its asymptote; `tan x` has none at any118 asymptote; `floor(x)` has no risers (a riser would be a 40-pixel vertical run in an integer column); asymptote guides119 are lighter than the curve; an aliased function is a band with a notice; hole markers, inequality fills and shaded120 regions render; dark appearance uses the dark tokens. `ScreenshotTests` renders the screen layouts.121- **Math** (`MathNotationTests.swift`): the layout tree of fractions, powers, radicals and matrices, spoken descriptions,122 measured geometry, and off-screen renders of formulas counted by inked pixels.123- **Statistics, Convert, Workspace** (`RenderTests.swift`, `RenderSmokeTests.swift`): every section or layout is rendered124 at phone and tablet sizes in light, dark and accessibility text sizes as a smoke test; the images are written to the125 temporary directory (Workspace: PNG files) for visual review.126127## Accessibility audit128129`testAccessibilityAuditOfTheCalculator` in `UITests/ORIXCalcUITests.swift` launches the Standard calculator and runs130`XCUIApplication.performAccessibilityAudit(for: [.dynamicType, .sufficientElementDescription, .hitRegion])`. The issue131handler ignores only the decorative separators inside the display, which are not interactive elements.132133## Math Camera recognition benchmark134135`RecognitionBenchmark.printedLinearExpressions` in136`Packages/ORIXMathCamera/Tests/ORIXMathCameraTests/RecognitionTests.swift` renders 30 printed expressions in three fonts137(Helvetica, Times New Roman, Menlo) with Core Text, runs `MathRecognizer.recognize` on each image and counts a hit when138the top candidate parses and evaluates to the same value as the source expression. It prints139140```141MATH-CAMERA-BENCHMARK total=… parsed=… correct=… rate=…142MATH-CAMERA-MISS <font>: <expression> → <raw> ⇒ <normalized>143```144145and asserts a floor of `rate >= 0.80`, which guards against regressions; the feature ships as a reviewed-entry aid, not as146an automatic solver. Read the measured rate from the test log (`build/logs/test-ORIXMathCamera.log`).147148## Performance149150- **Engine benchmark.** `cd Benchmarks && swift run -c release orix-bench` runs the expression, workspace and domain151 benchmarks (financial, statistics, graph, units) single-threaded and prints a Markdown table with the machine and build152 configuration. The recorded output is [`benchmark-results.md`](benchmark-results.md).153- **Graph timings.** `Packages/ORIXGraph/Tests/ORIXGraphTests/PerformanceTests.swift` prints `PERF|…` and `COUNT|…`154 lines and asserts time limits that are generous in debug builds and tight in release builds.155- **Symbolic bounds.** `BoundsAndPerformance` asserts size limits and a simplification time bound.156157## Regression tests158159Rule: **every bug fix adds a regression test.** The existing ones sit in the test target of the package that had the bug160and carry a comment saying where the bug was reported.161162Existing examples:163164| Test | Location | Bug it pins |165|---|---|---|166| `Regression.equationWithBuiltinOnTheLeft` | `Packages/ORIXExpression/Tests/ORIXExpressionTests/EngineTests.swift` | Reported by the graph engine: an equation whose left side is a built-in call (`sin(x) = cos(y)`) was read as a function definition |167| `Regression.longChainsOnASmallStack` | same | Reported by the symbolic engine: a sum of 900 terms overflowed the 512 KB stack of a secondary thread; also asserts that a 6000-term chain is refused with an error |168| `acklamCoefficientsAreTheRealOnes` | `Packages/ORIXStatistics/Tests/ORIXStatisticsTests/SpecialFunctionTests.swift` | Guards against a mistyped coefficient: the raw approximation must show its published 1.15e-9 bound |169| `bundledReferenceMatchesTheGeneratedFile` | `Packages/ORIXFinancial/Tests/ORIXFinancialTests/` | Guards against a stale bundled reference file |170| `RecognitionBenchmark` floor | `Packages/ORIXMathCamera/Tests/ORIXMathCameraTests/RecognitionTests.swift` | Guards the recognition rate |171172## Conventions173174- Tests state the tolerance they assert; the asserted tolerances are listed per engine in each `VALIDATION.md`.175- Tests that touch `AppModel` use an in-memory store and a unique `UserDefaults` suite, so they never read or write the176 user's data. The launch helper of the UI tests passes `-ORIXInMemory 1`.177- Engine tests do no networking; the ECB parser is tested against a bundled fixture.178- Error paths are tested as errors (`#expect(throws:)`), not as special values.179