SPB Git

Repository Public MIT

precision.md 2.7 KB

# Precision strategy

Kind of calculation Representation Consequence
Whole numbers Arbitrary-precision integers (BigInt) 2^200, 100!, nCr(100, 50) are exact, digit for digit
Decimals and fractions Exact rationals on BigInt 0.1 + 0.2 = 0.3 exactly; 1/3 + 1/3 + 1/3 = 1; a Hilbert-matrix determinant is exact
Roots and powers of exact values Exact when the result is rational (√(1/4), 27^(2/3)), otherwise double No false exactness
Special angles Exact table: sin 30° = 1/2, cos 90° = 0, tan 45° = 1; sin π = 0 by snapping the double nearest to k·π/2 tan 90° is a domain error, not 1.6e16
Transcendental functions IEEE-754 binary64 (libm) 15–16 significant digits, labelled "≈ floating point"
Complex numbers Complex<Double> (Swift Numerics), principal branches
Linear algebra Gauss–Jordan on typed values: exact for exact entries; partial pivoting for doubles; LAPACK dgeev for eigenvalues Residuals reported as verification checks
Money schedules Double for rates, Foundation Decimal for rounding each amount to the cent Lender statements can differ by a cent
Unit conversion Exact rational factors, 38-digit decimal results 1 mi = 1.609344 km exactly
Statistics Double, with special functions validated against SciPy / mpmath See Packages/ORIXStatistics/VALIDATION.md

# Limits (stated, not hidden)

  • Exact non-integer rationals are kept up to 8192 bits of numerator or denominator; beyond that the calculation continues in double precision with a precision note in the verification report.
  • Exact integers are limited to 4,000,000 bits (about 1.2 million digits); a larger result is an overflow error.
  • Factorials are exact up to 50000!.
  • Transcendental functions are not arbitrary precision. Foundation.Decimal is a 38-digit decimal float, not an arbitrary-precision type; it is used only where decimal rounding is the specification (money, unit display).
  • Doubles are displayed with at most 17 significant digits; the default display is 12, which hides the last-bit noise of binary floating point without altering the stored value.

# Never silently wrong

  • NaN and unexpected infinities are converted to errors at the point where they arise.
  • Display rounding is computed from the exact stored value and never written back.
  • Comparisons between an exact value and a double are done exactly (the double is converted to its exact rational value).
  • Conversion from an exact rational to a double is correctly rounded (round-to-nearest-even with a sticky bit, including subnormals), verified against known hard cases.
Files main
Menu

Theme