SPB Git

Repository Public MIT

reference-tests.md 15.6 KB

# Mathematical validation report

How the calculation engines are checked against independent references: where the references come from, how many cases there are, which tolerances the tests assert, and what is not independently validated.

This page describes the evidence in the repository. It is not a status page: run the suites with Scripts/test-all.sh (see docs/development/testing.md); the latest recorded results are kept in RELEASE-REPORT.md at the repository root. Per-engine detail is in each package's VALIDATION.md: Financial · Statistics · Units · Symbolic · Graph.

# Inventory

@Test functions counted in each package's Tests/ directory (grep -rE '@Test\b'); a parameterised or table-driven test counts once, however many reference cases it loops over. The last column is the figure stated in the package's VALIDATION.md, where there is one.

Engine package @Test @Suite Reference cases Stated in VALIDATION.md
ORIXNumerics 12 2 closed forms in the test source —
ORIXExpression 37 9 exact expected strings and closed forms in the test source —
ORIXFinancial 67 8 108 cases in reference.json + textbook and Excel-documentation values 67 tests in 8 suites
ORIXStatistics 85 15 5 343 special-function values, 10 164 distribution values, 8 + 4 data sets, 132 inference cases, 18 regression cases, 6 bootstrap cases 85 tests in 15 suites
ORIXUnits 60 11 841 conversions, 962 rate-convention values 60 tests in 11 suites
ORIXSymbolic 58 10 sympy / mpmath corpus (see table below) 58 tests in 10 suites
ORIXGraph 117 20 analytic values and analytically known discontinuities 117 tests in 20 suites
ORIXWorkspace 11 3 — (behavioural) —
ORIXMathCamera 3 2 30 printed expressions × 3 fonts —

Feature packages and the app are behavioural and rendering tests, not mathematical references; they are listed in docs/development/testing.md.

# How the references are produced

Engine Generator Libraries Output (in the package) Loaded by the tests from (in the package)
Financial Packages/ORIXFinancial/Reference/generate_reference.py mpmath (50 digits), numpy, scipy 1.17.1, decimal Reference/reference.json, console log Reference/reference_run.txt Tests/ORIXFinancialTests/Resources/reference.json (a test fails if the two copies differ)
Statistics Packages/ORIXStatistics/Reference/gen_*.py, reflib.py, driven by generate_all.sh mpmath 1.3.0 (50 digits; 60 for the regression solve), scipy 1.17.1, numpy 2.4.4, statsmodels 0.14.6, fractions, Python 3.14.4 (Reference/output/VERSIONS.txt) Reference/output/*.json Tests/ORIXStatisticsTests/ReferenceData/ (mirrored by the script)
Units Packages/ORIXUnits/Reference/generate_reference.py fractions.Fraction, decimal (60 digits) Reference/reference_conversions.tsv, reference_rates.tsv Tests/ORIXUnitsTests/ReferenceData.swift (embedded table)
Symbolic Packages/ORIXSymbolic/Reference/generate_reference.py sympy 1.14.0, mpmath 1.3.0 Reference/reference.json Tests/ORIXSymbolicTests/Resources/reference.json
Graph none: expectations are analytic — — test sources and Tests/ORIXGraphTests/Support.swift
Expression, Numerics none: exact strings and closed forms — — test sources

Each financial reference is computed by a different route from the Swift code (period-by-period compounding against the closed form, mpmath.polyroots against the scan + Brent, mpmath.diff against the analytic duration sums, vectorised numpy trees against scalar loops). mpmath, not scipy, is the primary reference for statistical functions because scipy is itself inaccurate in places at the accuracy tested; the disagreements found are recorded in the JSON under scipy_disagreements.

# Standard arithmetic

Packages/ORIXExpression/Tests/ORIXExpressionTests/EngineTests.swift, suite StandardArithmetic. Expected values are exact display strings.

Input Expected What it guards
25 × 4 100 basic arithmetic
2 ^ 3 ^ 2 512 right-associative power
-2 ^ 2 / (-2) ^ 2 −4 / 4 unary minus below power
6 ÷ 2(1 + 2) 9 implicit multiplication at the same level as ÷
0.1 + 0.2 0.3 exact decimal arithmetic
0.1 + 0.2 == 0.3 true exact comparison
1/3 + 1/3 + 1/3 1 exact rationals
200 + 10% 220 (contextual), 200.1 (strict) percent convention
100 + 5% + 5% 110.25 percentage of the running total
2^100 1267650600228229401496703205376 arbitrary-precision integers
25! 15511210043330985984000000 exact factorial
1e400 / 1e399 10 no intermediate overflow
1/0, 5 mod 0, 0^-1 CalcError.divisionByZero errors, not infinities
exp(1000) CalcError overflow is an error

# Scientific

Same file, suites Scientific, ComplexNumbers, Matrices, Formatting. Floating-point comparisons use a relative tolerance of 1e-12 unless a test states otherwise.

Input Expected What it guards
sin(30), cos(60) in degrees 0.5 exact special angles
sin(180), cos(90) in degrees; sin(pi), cos(pi/2) 0 snapping at multiples of π/2
tan(90) in degrees; tan(pi/2) CalcError pole is a domain error
sqrt(1/4) 0.5 exact root of an exact value
27^(2/3), (-8)^(1/3) 9, −2 exact rational powers
log(81, 3), log2(1/8) 4, −3 exact logarithms
nCr(100, 50) 100891344545564193334812497256 exact combinatorics
isprime(2^61 - 1) true number theory on large integers
sqrt(-1), ln(0), asin(2), 0/0 CalcError real-mode domain errors
sin(x)² + cos(x)² over sample points 1 within 1e-12 identities

# Financial

Tolerance is |obtained − expected| ≤ tol · max(1, |expected|) unless stated. "Python" means reference.json.

Case Source Expected Tolerance
TVM: 100 000, 6 % monthly, 360 payments → PMT standard textbook case −599.55 1e-9
TVM: 1 000 at 5 % for 10 years → FV standard textbook case 1 628.89 1e-9
BEGIN vs END: PV of 15 payments of 500 at 6 % property + Python BEGIN = END × 1.06 1e-13; 1e-12 vs Python
P/Y ≠ C/Y: TVMInputs(n: 300, iy: 5, pv: 300 000, py: 12, cy: 2) → PMT project brief; Python −1 744.81 exact cent
P/Y ≠ C/Y round trip: periodicRate(iy: 5, py: 12, cy: 2) → nominal property 5 % 1e-12
tvm set: 11 worksheets × 5 unknowns (END/BEGIN, P/Y ≠ C/Y, daily compounding, zero, negative and 1e-7 % rates) Python each solved variable 1e-9 (I/Y in %: 1e-8); periodic rate 1e-14
Canadian mortgage 300 000, 5 % semi-annual, 25 y, monthly project brief; Python 1 744.81 exact cent
Same loan, monthly compounding (US) Python 1 753.77 exact cent
loan_schedules set: 11 schedules (accelerated, prepayments, rate changes with both policies, override, 5-year term) Python decimal, ROUND_HALF_UP payments, totals, probe rows < 0.005
IRR of −1000, 2300, −1320 project brief; Python polynomial roots 10 % and 20 % (both reported) 1e-12
IRR of −1000, 2200, −1210 property double root found —
Multiple-rate TVM worksheet property refused with both candidates —
Excel PRICE example (5.75 %, 6.5 %, semi-annual, 30/360) Microsoft Excel documentation 94.63436 5e-6
Excel YIELD, DURATION, MDURATION, XIRR examples Microsoft Excel documentation 6.5 %; 10.9191453; 5.736; 0.373362535 5e-8; 5e-8; 5e-4; 5e-9
BSM call / put, S = 42, K = 40, r = 10 %, σ = 20 %, T = 0.5 Hull 4.76 / 0.81 2 decimals
American put, CRR 5 steps, S = K = 50, r = 10 %, σ = 40 %, T = 5/12 Hull 4.49 2 decimals
bsm set: 8 markets × call/put mpmath 50 digits price, Greeks, implied volatility price 1e-12·S + 1e-10·price; Greeks 1e-10·S; IV 1e-10 or price-noise/vega
crr set: 6 markets × call/put, 5 … 2 000 steps vectorised numpy European and American values 1e-10
bonds set: 10 dated bonds, every day-count convention mpmath, datetime, brentq accrued, prices, durations, convexity, yields 1e-12 prices and yields; 1e-11 durations / convexity

# Graphing

Expectations are analytic; the tests re-evaluate the function between the vertices the sampler returned (hiddenBreaks, spans, uncovered, maxScreenError, bounded). Window 800 × 600, x ∈ [−10, 10] unless noted.

Function Expected output Tolerance
1/x 2 polylines, pole at 0 with limits −∞ / +∞ pole located to 1e-12
tan(x) 7 polylines, 6 poles at (k + ½)π 1e-12
floor(x) 21 horizontal pieces, 20 jumps on the integers exact
sin(x)/x 1 polyline, removable discontinuity with hole (0, 1) —
(x² − 1)/(x − 1) removable at 1, hole value 2 1e-6
sqrt(x) starts at exactly (0, 0); undefined interval [−10, 0] boundary to 1e-15 off the grid
sin(1000x) no polyline, one band of 800 columns —
12 smooth functions polyline within half a pixel of the curve < 0.5 px
x² + y² = 25 at 400 × 400 one closed polyline, area 25π ± 0.05
(x − y)² = 0 nothing drawn, reliable == false with a reason —
roots of x³ − x, √2, cos x analytic values 1e-12 … 1e-13
∫₀^π sin = 2, ∫ e^(−x²) on [−8, 8] = √π analytic values 1e-12
∫ across a pole (1/x, tan x, …) refused with a message —

# Statistics

Case Source Asserted tolerance
Special functions, whole grid (shapes 1e-5 … 1e6, tails to 1e-300) mpmath 50 digits 1e-15 … 2e-12 relative, per function
14 distributions, 81 parameter sets: pdf/pmf, cdf, survival, quantiles mpmath 50 digits 1e-12 relative
Discrete quantiles (736 cases) mpmath exact integers
normQ(0.975) = 1.9599639845400538 mpmath 2e-15
t(5) ppf(0.975) = 2.5705818356363146 mpmath 1e-12
Nine quantile definitions, 1 080 values numpy.quantile 1e-13
One-sample, Welch, paired t; proportions; χ²; ANOVA scipy, statsmodels 1e-9 … 1e-10
Longley regression coefficients 60-digit mpmath solve (agrees with the NIST certified values) 1e-10
Simple regression, prediction intervals, curve fits scipy linregress, statsmodels, numpy 1e-9 … 1e-11
BCa interval on fixed replicates Efron's formulas in numpy / scipy 1e-9
SplitMix64 and xoshiro256** output vectors published vectors exact
Every sampler, n = 200 000, fixed seeds moments and empirical CDF 5 standard errors

# Units

Case Source Tolerance
1 mi → km = 1.609344; 1 in → m = 0.0254; 1 lb → kg = 0.45359237 Python Fraction from the primary definitions exact
1 gal (US) → L = 3.785411784; 1 gal (imp) → L = 4.54609 same exact
1 atm → Pa = 101325; 1 BTU → J = 1055.05585262; 1 nmi → m = 1852 same exact
590 terminating conversions same equal to the last digit
251 non-terminating conversions same 33 significant digits
Double path same 2 ulp
Round trip a → b → a property exact when the forward result terminated, else 1e-36
962 rate-convention values Python decimal, 60 digits 2e-13 relative

# Symbolic

Symbolic forms differ between systems, so comparisons are semantic: both results are evaluated at pseudo-random points and must agree to 1e-8 … 1e-10. Integrals are compared with mpmath quadrature of the integrand.

Area Cases Reference
Derivatives 83, 6 points each; 150 random trees sympy diff; Ridders derivative
Indefinite integrals 88, 5 intervals each (each also re-differentiated) mpmath quadrature
Definite integrals 20 + 20 structural mpmath quadrature
Simplification 136 corpus + 600 random trees value preservation, idempotence
Limits 37 + 16 structural sympy limit
Taylor coefficients 20 series (orders 3–7) sympy
Polynomial equations 18 sympy nroots
Factorisation 18 + 12 random sympy factor_list
Linear systems 9 (incl. 8 × 8 Hilbert) exact substitution
ODEs 7 tests closed forms with error bounds

Structural expectations cover the canonical simple cases (d/dx x² prints 2x).

# What is not independently validated

Assembled from the limitations stated in the VALIDATION.md files.

Independence of the references

  • The Python references for Financial and Units are independent implementations, written in the same work session from the same reading of the conventions. They catch arithmetic, precision and algorithm errors; they cannot catch a convention misunderstood in both places (the 30/360 US February rule, the month-end coupon rule, Actual/360 and Actual/365 Fixed period handling, the accelerated-payment definition; a wrong belief about what a unit is).
  • No QuantLib, numpy-financial, Bloomberg or physical-calculator cross-check was run. No value was checked against a BA II Plus or its manual. The five Excel documentation examples are the only third-party bond / XIRR figures, and they were quoted from memory of the Microsoft documentation, not re-downloaded.
  • NIST SP 811 guard values and most unit definitions were written from memory of the standard. Two sources were fetched and read: the CODATA 2022 atomic mass constant and Schedules II and III of the Canada Weights and Measures Act. ISO 4217 names and minor-unit digits were not diffed against the SIX list.
  • The published SplitMix64 / xoshiro256** vectors and the NIST Longley certified values were typed from memory; the tests rely on the 60-digit solve, and the agreement with the typed constants is asserted separately.
  • Graph expectations are analytic, not from an external plotting system.

Not covered

  • NIST StRD Norris is not included; degree-10 polynomial problems (Filip, Wampler 4–5) were not attempted.
  • No formal random-number battery (TestU01, PractRand) was run. The end-to-end bootstrap comparison is statistical (tolerance 6 % of the interval width).
  • Excel PRICE / YIELD behaviour with one coupon or less remaining (simple-interest formula) is not reproduced and not validated.
  • The ECB parser is tested against a document in the real structure with made-up rates, not against a live download.
  • LaTeX output of symbolic results was not checked beyond what the printer guarantees; behaviour with user-defined functions inside symbolic expressions is not validated.
  • Symbolic verification by sampling is evidence, not proof (1e-7 relative on up to 10 points).
  • Graph detection of sub-pixel features, of zero sets without a sign change and of narrow poles before integration is heuristic; see the limits in Packages/ORIXGraph/VALIDATION.md §5.
  • The Math Camera benchmark uses rendered images of printed expressions, not photographs.

Platform

  • The engine VALIDATION.md records state that the engine tests were run with swift test on macOS (arm64) only and were not executed on an iOS simulator or device. Nothing was run on a physical device. Performance figures come from one Apple Silicon Mac.
Files main
Menu

Theme