Mathematical validation report
How the calculation engines are checked against independent references: where the references come from, how many cases there are, which tolerances the tests assert, and what is not independently validated.
This page describes the evidence in the repository. It is not a status page: run the suites with Scripts/test-all.sh
(see docs/development/testing.md); the latest recorded results are kept in
RELEASE-REPORT.md at the repository root. Per-engine detail is in each package's VALIDATION.md:
Financial · Statistics ·
Units · Symbolic ·
Graph.
Inventory
@Test functions counted in each package's Tests/ directory (grep -rE '@Test\b'); a parameterised or table-driven test
counts once, however many reference cases it loops over. The last column is the figure stated in the package's
VALIDATION.md, where there is one.
| Engine package | @Test |
@Suite |
Reference cases | Stated in VALIDATION.md |
|---|---|---|---|---|
ORIXNumerics |
12 | 2 | closed forms in the test source | — |
ORIXExpression |
37 | 9 | exact expected strings and closed forms in the test source | — |
ORIXFinancial |
67 | 8 | 108 cases in reference.json + textbook and Excel-documentation values |
67 tests in 8 suites |
ORIXStatistics |
85 | 15 | 5 343 special-function values, 10 164 distribution values, 8 + 4 data sets, 132 inference cases, 18 regression cases, 6 bootstrap cases | 85 tests in 15 suites |
ORIXUnits |
60 | 11 | 841 conversions, 962 rate-convention values | 60 tests in 11 suites |
ORIXSymbolic |
58 | 10 | sympy / mpmath corpus (see table below) | 58 tests in 10 suites |
ORIXGraph |
117 | 20 | analytic values and analytically known discontinuities | 117 tests in 20 suites |
ORIXWorkspace |
11 | 3 | — (behavioural) | — |
ORIXMathCamera |
3 | 2 | 30 printed expressions × 3 fonts | — |
Feature packages and the app are behavioural and rendering tests, not mathematical references; they are listed in
docs/development/testing.md.
How the references are produced
| Engine | Generator | Libraries | Output (in the package) | Loaded by the tests from (in the package) |
|---|---|---|---|---|
| Financial | Packages/ORIXFinancial/Reference/generate_reference.py |
mpmath (50 digits), numpy, scipy 1.17.1, decimal |
Reference/reference.json, console log Reference/reference_run.txt |
Tests/ORIXFinancialTests/Resources/reference.json (a test fails if the two copies differ) |
| Statistics | Packages/ORIXStatistics/Reference/gen_*.py, reflib.py, driven by generate_all.sh |
mpmath 1.3.0 (50 digits; 60 for the regression solve), scipy 1.17.1, numpy 2.4.4, statsmodels 0.14.6, fractions, Python 3.14.4 (Reference/output/VERSIONS.txt) |
Reference/output/*.json |
Tests/ORIXStatisticsTests/ReferenceData/ (mirrored by the script) |
| Units | Packages/ORIXUnits/Reference/generate_reference.py |
fractions.Fraction, decimal (60 digits) |
Reference/reference_conversions.tsv, reference_rates.tsv |
Tests/ORIXUnitsTests/ReferenceData.swift (embedded table) |
| Symbolic | Packages/ORIXSymbolic/Reference/generate_reference.py |
sympy 1.14.0, mpmath 1.3.0 | Reference/reference.json |
Tests/ORIXSymbolicTests/Resources/reference.json |
| Graph | none: expectations are analytic | — | — | test sources and Tests/ORIXGraphTests/Support.swift |
| Expression, Numerics | none: exact strings and closed forms | — | — | test sources |
Each financial reference is computed by a different route from the Swift code (period-by-period compounding against the
closed form, mpmath.polyroots against the scan + Brent, mpmath.diff against the analytic duration sums, vectorised
numpy trees against scalar loops). mpmath, not scipy, is the primary reference for statistical functions because scipy
is itself inaccurate in places at the accuracy tested; the disagreements found are recorded in the JSON under
scipy_disagreements.
Standard arithmetic
Packages/ORIXExpression/Tests/ORIXExpressionTests/EngineTests.swift, suite StandardArithmetic. Expected values are
exact display strings.
| Input | Expected | What it guards |
|---|---|---|
25 × 4 |
100 |
basic arithmetic |
2 ^ 3 ^ 2 |
512 |
right-associative power |
-2 ^ 2 / (-2) ^ 2 |
−4 / 4 |
unary minus below power |
6 ÷ 2(1 + 2) |
9 |
implicit multiplication at the same level as ÷ |
0.1 + 0.2 |
0.3 |
exact decimal arithmetic |
0.1 + 0.2 == 0.3 |
true |
exact comparison |
1/3 + 1/3 + 1/3 |
1 |
exact rationals |
200 + 10% |
220 (contextual), 200.1 (strict) |
percent convention |
100 + 5% + 5% |
110.25 |
percentage of the running total |
2^100 |
1267650600228229401496703205376 |
arbitrary-precision integers |
25! |
15511210043330985984000000 |
exact factorial |
1e400 / 1e399 |
10 |
no intermediate overflow |
1/0, 5 mod 0, 0^-1 |
CalcError.divisionByZero |
errors, not infinities |
exp(1000) |
CalcError |
overflow is an error |
Scientific
Same file, suites Scientific, ComplexNumbers, Matrices, Formatting. Floating-point comparisons use a relative
tolerance of 1e-12 unless a test states otherwise.
| Input | Expected | What it guards |
|---|---|---|
sin(30), cos(60) in degrees |
0.5 |
exact special angles |
sin(180), cos(90) in degrees; sin(pi), cos(pi/2) |
0 |
snapping at multiples of π/2 |
tan(90) in degrees; tan(pi/2) |
CalcError |
pole is a domain error |
sqrt(1/4) |
0.5 |
exact root of an exact value |
27^(2/3), (-8)^(1/3) |
9, −2 |
exact rational powers |
log(81, 3), log2(1/8) |
4, −3 |
exact logarithms |
nCr(100, 50) |
100891344545564193334812497256 |
exact combinatorics |
isprime(2^61 - 1) |
true |
number theory on large integers |
sqrt(-1), ln(0), asin(2), 0/0 |
CalcError |
real-mode domain errors |
sin(x)² + cos(x)² over sample points |
1 within 1e-12 | identities |
Financial
Tolerance is |obtained − expected| ≤ tol · max(1, |expected|) unless stated. "Python" means reference.json.
| Case | Source | Expected | Tolerance |
|---|---|---|---|
| TVM: 100 000, 6 % monthly, 360 payments → PMT | standard textbook case | −599.55 | 1e-9 |
| TVM: 1 000 at 5 % for 10 years → FV | standard textbook case | 1 628.89 | 1e-9 |
| BEGIN vs END: PV of 15 payments of 500 at 6 % | property + Python | BEGIN = END × 1.06 | 1e-13; 1e-12 vs Python |
P/Y ≠ C/Y: TVMInputs(n: 300, iy: 5, pv: 300 000, py: 12, cy: 2) → PMT |
project brief; Python | −1 744.81 | exact cent |
P/Y ≠ C/Y round trip: periodicRate(iy: 5, py: 12, cy: 2) → nominal |
property | 5 % | 1e-12 |
tvm set: 11 worksheets × 5 unknowns (END/BEGIN, P/Y ≠ C/Y, daily compounding, zero, negative and 1e-7 % rates) |
Python | each solved variable | 1e-9 (I/Y in %: 1e-8); periodic rate 1e-14 |
| Canadian mortgage 300 000, 5 % semi-annual, 25 y, monthly | project brief; Python | 1 744.81 | exact cent |
| Same loan, monthly compounding (US) | Python | 1 753.77 | exact cent |
loan_schedules set: 11 schedules (accelerated, prepayments, rate changes with both policies, override, 5-year term) |
Python decimal, ROUND_HALF_UP |
payments, totals, probe rows | < 0.005 |
| IRR of −1000, 2300, −1320 | project brief; Python polynomial roots | 10 % and 20 % (both reported) | 1e-12 |
| IRR of −1000, 2200, −1210 | property | double root found | — |
| Multiple-rate TVM worksheet | property | refused with both candidates | — |
Excel PRICE example (5.75 %, 6.5 %, semi-annual, 30/360) |
Microsoft Excel documentation | 94.63436 | 5e-6 |
Excel YIELD, DURATION, MDURATION, XIRR examples |
Microsoft Excel documentation | 6.5 %; 10.9191453; 5.736; 0.373362535 | 5e-8; 5e-8; 5e-4; 5e-9 |
| BSM call / put, S = 42, K = 40, r = 10 %, σ = 20 %, T = 0.5 | Hull | 4.76 / 0.81 | 2 decimals |
| American put, CRR 5 steps, S = K = 50, r = 10 %, σ = 40 %, T = 5/12 | Hull | 4.49 | 2 decimals |
bsm set: 8 markets × call/put |
mpmath 50 digits | price, Greeks, implied volatility | price 1e-12·S + 1e-10·price; Greeks 1e-10·S; IV 1e-10 or price-noise/vega |
crr set: 6 markets × call/put, 5 … 2 000 steps |
vectorised numpy | European and American values | 1e-10 |
bonds set: 10 dated bonds, every day-count convention |
mpmath, datetime, brentq |
accrued, prices, durations, convexity, yields | 1e-12 prices and yields; 1e-11 durations / convexity |
Graphing
Expectations are analytic; the tests re-evaluate the function between the vertices the sampler returned
(hiddenBreaks, spans, uncovered, maxScreenError, bounded). Window 800 × 600, x ∈ [−10, 10] unless noted.
| Function | Expected output | Tolerance |
|---|---|---|
1/x |
2 polylines, pole at 0 with limits −∞ / +∞ | pole located to 1e-12 |
tan(x) |
7 polylines, 6 poles at (k + ½)π | 1e-12 |
floor(x) |
21 horizontal pieces, 20 jumps on the integers | exact |
sin(x)/x |
1 polyline, removable discontinuity with hole (0, 1) | — |
(x² − 1)/(x − 1) |
removable at 1, hole value 2 | 1e-6 |
sqrt(x) |
starts at exactly (0, 0); undefined interval [−10, 0] | boundary to 1e-15 off the grid |
sin(1000x) |
no polyline, one band of 800 columns | — |
| 12 smooth functions | polyline within half a pixel of the curve | < 0.5 px |
x² + y² = 25 at 400 × 400 |
one closed polyline, area 25π | ± 0.05 |
(x − y)² = 0 |
nothing drawn, reliable == false with a reason |
— |
roots of x³ − x, √2, cos x |
analytic values | 1e-12 … 1e-13 |
∫₀^π sin = 2, ∫ e^(−x²) on [−8, 8] = √π |
analytic values | 1e-12 |
∫ across a pole (1/x, tan x, …) |
refused with a message | — |
Statistics
| Case | Source | Asserted tolerance |
|---|---|---|
| Special functions, whole grid (shapes 1e-5 … 1e6, tails to 1e-300) | mpmath 50 digits | 1e-15 … 2e-12 relative, per function |
| 14 distributions, 81 parameter sets: pdf/pmf, cdf, survival, quantiles | mpmath 50 digits | 1e-12 relative |
| Discrete quantiles (736 cases) | mpmath | exact integers |
normQ(0.975) = 1.9599639845400538 |
mpmath | 2e-15 |
t(5) ppf(0.975) = 2.5705818356363146 |
mpmath | 1e-12 |
| Nine quantile definitions, 1 080 values | numpy.quantile |
1e-13 |
| One-sample, Welch, paired t; proportions; χ²; ANOVA | scipy, statsmodels | 1e-9 … 1e-10 |
| Longley regression coefficients | 60-digit mpmath solve (agrees with the NIST certified values) | 1e-10 |
| Simple regression, prediction intervals, curve fits | scipy linregress, statsmodels, numpy |
1e-9 … 1e-11 |
| BCa interval on fixed replicates | Efron's formulas in numpy / scipy | 1e-9 |
| SplitMix64 and xoshiro256** output vectors | published vectors | exact |
| Every sampler, n = 200 000, fixed seeds | moments and empirical CDF | 5 standard errors |
Units
| Case | Source | Tolerance |
|---|---|---|
1 mi → km = 1.609344; 1 in → m = 0.0254; 1 lb → kg = 0.45359237 |
Python Fraction from the primary definitions |
exact |
1 gal (US) → L = 3.785411784; 1 gal (imp) → L = 4.54609 |
same | exact |
1 atm → Pa = 101325; 1 BTU → J = 1055.05585262; 1 nmi → m = 1852 |
same | exact |
| 590 terminating conversions | same | equal to the last digit |
| 251 non-terminating conversions | same | 33 significant digits |
Double path |
same | 2 ulp |
| Round trip a → b → a | property | exact when the forward result terminated, else 1e-36 |
| 962 rate-convention values | Python decimal, 60 digits |
2e-13 relative |
Symbolic
Symbolic forms differ between systems, so comparisons are semantic: both results are evaluated at pseudo-random points and must agree to 1e-8 … 1e-10. Integrals are compared with mpmath quadrature of the integrand.
| Area | Cases | Reference |
|---|---|---|
| Derivatives | 83, 6 points each; 150 random trees | sympy diff; Ridders derivative |
| Indefinite integrals | 88, 5 intervals each (each also re-differentiated) | mpmath quadrature |
| Definite integrals | 20 + 20 structural | mpmath quadrature |
| Simplification | 136 corpus + 600 random trees | value preservation, idempotence |
| Limits | 37 + 16 structural | sympy limit |
| Taylor coefficients | 20 series (orders 3–7) | sympy |
| Polynomial equations | 18 | sympy nroots |
| Factorisation | 18 + 12 random | sympy factor_list |
| Linear systems | 9 (incl. 8 × 8 Hilbert) | exact substitution |
| ODEs | 7 tests | closed forms with error bounds |
Structural expectations cover the canonical simple cases (d/dx x² prints 2x).
What is not independently validated
Assembled from the limitations stated in the VALIDATION.md files.
Independence of the references
- The Python references for Financial and Units are independent implementations, written in the same work session from the same reading of the conventions. They catch arithmetic, precision and algorithm errors; they cannot catch a convention misunderstood in both places (the 30/360 US February rule, the month-end coupon rule, Actual/360 and Actual/365 Fixed period handling, the accelerated-payment definition; a wrong belief about what a unit is).
- No QuantLib, numpy-financial, Bloomberg or physical-calculator cross-check was run. No value was checked against a BA II Plus or its manual. The five Excel documentation examples are the only third-party bond / XIRR figures, and they were quoted from memory of the Microsoft documentation, not re-downloaded.
- NIST SP 811 guard values and most unit definitions were written from memory of the standard. Two sources were fetched and read: the CODATA 2022 atomic mass constant and Schedules II and III of the Canada Weights and Measures Act. ISO 4217 names and minor-unit digits were not diffed against the SIX list.
- The published SplitMix64 / xoshiro256** vectors and the NIST Longley certified values were typed from memory; the tests rely on the 60-digit solve, and the agreement with the typed constants is asserted separately.
- Graph expectations are analytic, not from an external plotting system.
Not covered
- NIST StRD Norris is not included; degree-10 polynomial problems (Filip, Wampler 4–5) were not attempted.
- No formal random-number battery (TestU01, PractRand) was run. The end-to-end bootstrap comparison is statistical (tolerance 6 % of the interval width).
- Excel
PRICE/YIELDbehaviour with one coupon or less remaining (simple-interest formula) is not reproduced and not validated. - The ECB parser is tested against a document in the real structure with made-up rates, not against a live download.
- LaTeX output of symbolic results was not checked beyond what the printer guarantees; behaviour with user-defined functions inside symbolic expressions is not validated.
- Symbolic verification by sampling is evidence, not proof (1e-7 relative on up to 10 points).
- Graph detection of sub-pixel features, of zero sets without a sign change and of narrow poles before integration is
heuristic; see the limits in
Packages/ORIXGraph/VALIDATION.md§5. - The Math Camera benchmark uses rendered images of printed expressions, not photographs.
Platform
- The engine
VALIDATION.mdrecords state that the engine tests were run withswift teston macOS (arm64) only and were not executed on an iOS simulator or device. Nothing was run on a physical device. Performance figures come from one Apple Silicon Mac.