Reference tests
How the numerical and behavioural correctness of ORIX Sheet is established, what the references are, how many cases each one contributes, the tolerances asserted, how to regenerate the fixtures — and where the method stops.
Related documents: numerical-precision.md, financial-conventions.md, ../compatibility/formulas.md (per-function validation source), ../compatibility/excel.md, ../development/testing.md (how to run the suites).
1. Principle
No result is accepted because the Swift code says so. Each computed value is compared with a number
produced by a program that shares no code with ORIX Sheet: another spreadsheet (LibreOffice), a
scientific library (SciPy, mpmath, NumPy, statsmodels, scikit-learn, pandas), exact decimal arithmetic
(Python decimal), a file-format library (openpyxl), values published as computed by Microsoft Excel,
or the worked examples of Microsoft's function reference typed by hand.
The reference values are generated once by scripts kept next to the tests, committed as fixtures, and
replayed by swift test without Python, LibreOffice or a network connection.
flowchart LR
G["Generator scripts<br/>Tests/*/Generators/*.py"] --> LO["LibreOffice headless<br/>recalculation"]
G --> PY["Python references<br/>SciPy, mpmath, NumPy,<br/>statsmodels, pandas, decimal"]
XL["Excel-computed public data<br/>(ExcelFinancialFunctions)"] --> CV["gen_excel_reference.py"]
LO --> FX["Committed fixtures<br/>Tests/*/Fixtures/"]
PY --> FX
CV --> FX
DOC["Microsoft documentation<br/>examples, typed by hand"] --> HW["Hand-written Swift tests"]
FX --> ST["swift test<br/>fixture replay"]
HW --> ST
ST --> EN["ORIX Sheet engine<br/>and libraries"]
SVG fallback:
2. The honest limits, first
- Microsoft Excel was never driven. Excel is installed on the build machine
(
docs/research/technology-evaluation.md), but no test, generator or script launches it, opens a file in it or reads a value from it. Every package document repeats this: "Nothing in this document comes from running Excel" (Packages/FunctionsNumeric/COMPATIBILITY.md), "No result is validated against a live Excel" (Packages/FunctionsText/COMPATIBILITY.md), "Microsoft Excel itself was not run" (Packages/SheetIO/COMPATIBILITY.md). - The only values that come from Excel are the public test data of the ExcelFinancialFunctions project (results computed by Excel for large argument matrices, Apache License 2.0), used for the financial functions and YEARFRAC. They cover valid inputs only (plus PDURATION / RRI failures); error kinds for invalid arguments rest on documentation and on LibreOffice returning an error.
- Excel's behaviour elsewhere is taken from Microsoft's documentation. Where the documentation is silent, the behaviour is either LibreOffice's (when the two programs are known to agree) or marked "not verified" in the package document and in ../compatibility/excel.md.
- LibreOffice is not Excel. It disagrees with Excel's documented behaviour in several thousand of the generated cases (section 6). Those cases are not silently dropped: the fixture stores the Excel-documented value, a note and the LibreOffice value.
- LibreOffice error codes are not compared. Its XLSX export reports most errors as
#VALUE!and logical values as 1/0, so fixtures check "is an error" (exact kind only for#N/Aand#DIV/0!inFunctionsLookup); exact Excel error kinds are asserted by hand-written tests. - XLSX files written by ORIX Sheet were opened by openpyxl and LibreOffice only. Nothing here is a claim that Excel opens them without a repair prompt.
- A reference test proves agreement on the tested inputs, not correctness everywhere. The grids are described in the package documents; outside them the code is unverified.
3. Reference sources
| Reference | Version stated in the package documents | Used for |
|---|---|---|
| LibreOffice Calc, headless recalculation of generated workbooks | 26.8 / 26.8.2 (en-US profile) | spreadsheet semantics of functions (coercion, blanks, criteria, ranges, arrays, text), number-format rendering, XLSX/ODS interoperability, template recalculation |
| Microsoft Excel-computed public data | ExcelFinancialFunctions test data (fsprojects, Copyright 2010-2013, Apache License 2.0) | financial functions and YEARFRAC |
| SciPy + mpmath | SciPy 1.17 (1.17.1), mpmath 1.3 at 40 digits (up to 400 for arbitration) | distributions, quantiles, special functions, Bessel, trigonometry, unit conversions, root finding, statistics |
| NumPy | 2.4 (2.4.4) | descriptive statistics, percentiles, histograms, linear algebra, portfolio measures |
| statsmodels | 0.14.6 | OLS / WLS / panel / logit regression blocks, LINEST statistics, prediction intervals, time series |
| scikit-learn | 1.6 | multivariate methods of SheetAnalytics (stated in Packages/SheetAnalytics/VALIDATION.md) |
arch |
8 | GARCH |
Python decimal |
Python 3.14 standard library | ROUND family on the 15-digit decimal reading, MOD, QUOTIENT, amortization schedules to the cent (60-digit context) |
Python cmath, datetime, dateutil, re, str |
standard library + python-dateutil | complex functions, date functions, regular expressions and text functions LibreOffice lacks |
| pandas (+ numpy, pyarrow; no DuckDB) | pandas 3.0 (3.0.2) | data engine: aggregates, profiles, transforms, joins, pivot cubes; CSV round trips of SheetIO |
| openpyxl | 3.1.5 | independent reader of XLSX files written by ORIX Sheet, and generator of XLSX fixtures read by ORIX Sheet |
| matplotlib | version not stated | tick locator, box-plot statistics, colour maps, contour lengths (chart engine) |
| Microsoft function reference | worked examples typed by hand | documented results to the printed digits, documented error codes |
Source: Packages/FunctionsNumeric/COMPATIBILITY.md, Packages/FunctionsText/COMPATIBILITY.md, Packages/FunctionsLookup/COMPATIBILITY.md, Packages/SheetAnalytics/VALIDATION.md, Packages/SheetFormat/COMPATIBILITY.md, Packages/SheetIO/COMPATIBILITY.md, Packages/SheetData/README.md, Packages/SheetCharts/README.md, docs/research/technology-evaluation.md (installed versions).
4. Case counts per package
The counts in the "Stated" column are copied from the package documents. The "Counted" column was obtained on 2026-11-21 by loading the committed fixture files with Python and counting their cases; it is there to show that the documents and the files agree.
| Package | Fixture(s) | Stated | Counted in the files |
|---|---|---|---|
| FunctionsNumeric | 24 JSON files in Tests/FunctionsNumericTests/Fixtures/ |
15 378 | 15 378 |
| FunctionsText | function_references.json |
7 951 | 7 951 |
| FunctionsLookup | excel_reference.json, financial.json, securities.json, lookup_lo.json, arrays.json |
86 965 formula cases + 1 397 YEARFRAC rows | 86 965 + 1 397 |
| SheetFormat | libreoffice_formats.json |
5 085 | 5 085 |
Source: package documents named in section 3; counts measured with a Python script over Packages/*/Tests/*/Fixtures/*.json on 2026-11-21.
FunctionsNumeric (296 functions)
| Source | Cases | What it vouches for |
|---|---|---|
LibreOffice 26.8 (headless recalculation of generated .xlsx workbooks, en-US locale) |
4 687 | spreadsheet semantics: coercion, blanks, criteria, ranges, array results, text results |
| scipy 1.17 + mpmath 1.3 (40 digits) | 5 440 | distributions, quantiles, special functions, Bessel, trigonometry, unit conversions (scipy.constants) |
Python decimal |
4 082 | ROUND / ROUNDUP / ROUNDDOWN / TRUNC on the 15-digit decimal reading, MOD, QUOTIENT |
| numpy + scipy.stats | 606 | descriptive statistics, percentiles, ranks, correlation, t / F / z / χ² tests |
| statsmodels OLS + numpy.linalg + mpmath | 143 | LINEST / LOGEST statistics blocks, TREND, GROWTH, MINVERSE, MDETERM, MMULT |
Python cmath |
420 | complex-number functions |
| Reference corpus (JSON fixtures) | 15 378 | |
| Microsoft documentation examples (hand-written) | ≈ 370 | ExcelDocumentationTests.swift: the examples of Excel's help pages, to the printed digits |
| Hand-written edge cases | ≈ 500 | EdgeCaseTests.swift: 201 Excel error codes, logical values, arrays, blanks, whole columns, hidden rows |
Source: Packages/FunctionsNumeric/COMPATIBILITY.md ("How the functions are validated"). The per-source totals were recounted from the source field of the 24 fixture files on 2026-11-21 and match (libreoffice 4 687, scipy 5 440, decimal 4 082, numpy 606, statsmodels 143, cmath 420).
The per-function breakdown (LibreOffice cases, independent cases, hand-written cases for each of the 296
functions) is the "Function table" of Packages/FunctionsNumeric/COMPATIBILITY.md, reproduced in
../compatibility/formulas.md.
FunctionsText (64 functions: 41 text, 23 date and time)
| Source tag | Cases | Meaning |
|---|---|---|
lo |
3 565 | formula recalculated by headless LibreOffice 26.8 (en-US profile) |
lo+py |
1 790 | LibreOffice result that also equals an independent Python reference (datetime, dateutil; the generator aborts on any disagreement): YEAR, MONTH, DAY, HOUR, MINUTE, SECOND, WEEKDAY, WEEKNUM, ISOWEEKNUM, EDATE, EOMONTH, DAYS, NETWORKDAYS, WORKDAY |
py |
2 362 | Python reference (re, str) for the functions LibreOffice does not implement: VALUETOTEXT, ARRAYTOTEXT, REGEXTEST, REGEXEXTRACT, REGEXREPLACE |
excel |
234 | LibreOffice deviates from Excel's documented behaviour; the Excel value is stored with a note, the LibreOffice value is kept in lo |
| Total | 7 951 |
Source: Packages/FunctionsText/COMPATIBILITY.md ("Validation"); the four totals were recounted from the src field of function_references.json on 2026-11-21 and match; the file covers 64 distinct function names.
About 310 hand-written expectations add Microsoft's documented examples, array lifting, ranges, blanks, errors, UTF-16 positions, the 1900 leap bug, the 1904 system and the fr-CA locale. The reference test also fails when a registered function has no reference case.
FunctionsLookup (81 functions: 10 lookup and reference, 22 dynamic-array, 49 financial)
| Tag | Source | Cases |
|---|---|---|
| XL | Values computed by Microsoft Excel: test data of ExcelFinancialFunctions, converted by Generators/gen_excel_reference.py into Fixtures/excel_reference.json (dates → serials, enumerations → Excel codes; nothing else changed, nothing thinned) |
68 075 + 1 397 YEARFRAC |
| LO | LibreOffice Calc 26.8.2 headless recalculation of generated workbooks (Generators/lo_harness.py) |
14 223 |
| PY | Independent Python computations (Generators/pyref.py, gen_arrays.py): closed forms, period-by-period simulations, explicit cash-flow discounting, scipy.optimize.brentq |
17 874 (13 207 of them also confirmed by LO) |
| DOC | Worked examples of Microsoft's function reference, typed by hand in FinancialTests.swift |
90 |
| HW | Hand-written Excel-semantics tests (LookupTests, DynamicArrayTests, FinancialTests, PublicAPITests) |
≈ 500 assertions |
Source: Packages/FunctionsLookup/COMPATIBILITY.md ("Reference sources"). File sizes counted on 2026-11-21: excel_reference.json 68 075 cases + 1 397 YEARFRAC rows, securities.json 13 153, financial.json 4 591, lookup_lo.json 847, arrays.json 299 (sum of the four formula files and the Excel file: 86 965).
The number of Excel-computed cases per financial function (for example PRICE 10 982, DURATION 5 492,
IPMT 4 552, VDB 2 546, RATE 8, IRR 11) is in the "Financial" table of
Packages/FunctionsLookup/COMPATIBILITY.md. Three gaps are stated there and matter:
- XNPV has no Excel data set (LibreOffice + Python + documentation example).
- YIELD: the Excel set covers negative yields (441 cases); positive yields rest on PRICE being inverted (10 982 Excel cases for PRICE), on LibreOffice + Python and on the documentation example.
- ACCRINT: the Excel data covers settlement ≤
first_interestwithcalc_methodTRUE only.
SheetFormat (number formats)
- 5 085
=TEXT(value, format)results computed by headless LibreOffice 26.8 (70 numeric formats × 44 values, 61 date/time formats × 32 serials, text cases). 4 246 are byte-identical; 839 differ in one of 13 documented ways, each checked by a narrow rule inExcelDeviations.swiftso that any other difference fails the test. - About 340 hand-written expectations (
EdgeTests.swift). - A fuzz test renders 20 000 random format codes against 12 values in two locales (no crash, bounded General width).
Source: Packages/SheetFormat/COMPATIBILITY.md ("Validation").
SheetAnalytics (statistics, finance models, what-if, audit)
Packages/SheetAnalytics/VALIDATION.md lists every public routine with its reference, the tolerance
asserted and its status. It states 305 tests and about 2 600 #expect sites, many inside loops over
fixture cases; the document does not give a number of reference cases, so none is quoted here.
| Section of VALIDATION.md | Rows in the table | References named |
|---|---|---|
| Statistics — core, distributions, descriptive statistics, inference | 51 | scipy.special, scipy.stats, closed forms and round trips where SciPy is unreliable, pure-Python xoshiro256** reference |
| Statistics — regression and econometrics | 27 | statsmodels (OLS, WLS, robust errors, panel, Logit), numpy |
| Statistics — time series, multivariate, least squares | 29 | statsmodels, scikit-learn, arch, SciPy |
| Statistics — resampling; WhatIf target | 25 | scipy.stats.bootstrap with the same index matrix, hand-derived exact expectations |
| FinanceModels | 30 | independent Python formulas, Python decimal schedules, scipy.optimize.brentq, numpy.roots, SLSQP, Excel documentation examples |
| Audit — Model Auditor and Formula Intelligence | 12 | hand-derived exact expectations on workbooks built in the tests |
| Audit — conditional formatting, validation, tables | 20 | hand-derived exact expectations |
Source: Packages/SheetAnalytics/VALIDATION.md; row counts obtained by counting the table lines of each section (header and separator lines excluded) on 2026-11-21. Rows include the entries marked "not shipped".
Fixtures: Tests/StatisticsTests/Fixtures/ (core.json, descriptive_inference.json,
regression.json, timeseries.json, multivariate.json, resampling.json, tukey.json) and
Tests/FinanceModelsTests/Fixtures/finance.json.
SheetData (data engine)
Tests/SheetDataTests/Generators/generate_fixtures.py (pandas, numpy, pyarrow — no DuckDB) generates
seeded synthetic files (CSV, Parquet, JSON Lines, SQLite, Arrow, CSV dialects, hostile column names) and
expected.json: aggregates, profiles, the expected table of every transform scenario, joins of each
kind, and pivot cubes computed with groupby at every (row prefix, column prefix) level, show-as
variants, sort orders and drill-down row sets. The Swift tests compare the engine's output with those
references cell by cell.
Source: Packages/SheetData/README.md ("Tests").
SheetIO (file formats)
There is no numeric corpus: the references are other programs reading what ORIX Sheet writes and
writing what ORIX Sheet reads. Tests/SheetIOTests/Fixtures/ holds files produced by openpyxl
(features_openpyxl.xlsx, charts_openpyxl.xlsx, dates1904_openpyxl.xlsx), by LibreOffice
(features_lo.xlsx, features_lo.ods), a hand-built raw XLSX (raw_features.xlsx) and a macro workbook
(macro.xlsm). InteropTests.swift runs openpyxl, pandas and LibreOffice on files written by the
tests. The feature-by-feature "Verified by" column is in Packages/SheetIO/COMPATIBILITY.md and in
../compatibility/formats.md.
SheetCharts (chart numerics)
Tests/SheetChartsTests/Fixtures/reference.json and colormaps.json are generated by gen_fixtures.py
and gen_colormaps.py (numpy 2.4.4, scipy 1.17.1, matplotlib): tick positions
(matplotlib.ticker.MaxNLocator), histogram bins (numpy.histogram_bin_edges), quantile types
(numpy.quantile), box-plot statistics (matplotlib.cbook.boxplot_stats), kernel density
(scipy.stats.gaussian_kde), trendlines, stacking, contour lengths and colour maps (within 4/255 of
matplotlib). The treemap is covered by property tests, not by a reference; the 3-D surface has no
numerical reference.
Source: Packages/SheetCharts/README.md ("Statistical conventions").
SheetTemplates
Tests/SheetTemplatesTests/Generators/gen_expected.py recomputes the key outputs of every
template with numpy / scipy and closed forms ("never with a spreadsheet engine") into
Fixtures/expected.json, which NumericTests.swift compares with the calculated workbooks; and
LibreOfficeTests.swift exports every template with poisoned cached values, lets headless LibreOffice
recalculate it and compares every formula cell and spilled cell at a relative tolerance of 1e-6. The
last run in build/test-logs/SheetTemplates.log compared 18,714 cells in 30 workbooks (15 templates ×
2 languages) with 0 mismatches. One waiver is recorded in the test: 22 cells of the French
sales-tracking template (Sommaire!I10:K17, a SORT of accented product names) are not compared. The
waiver was written when ORIX Sheet ordered text by code point; text ordering now uses a fixed
collation (numerical-precision.md) but the waiver is still in the test, so
those cells remain unchecked against LibreOffice. See ../templates.md.
Core engine (SheetCore, SheetEngine) and intelligence
Packages/SheetCore/Tests/SheetCoreTests/CoreTests.swift(addresses, parser, semantics, recalculation, editing, input) andPackages/SheetEngine/Tests/SheetEngineTests/EngineTests.swiftare hand-written expectations. There is no external reference corpus for the parser, the dependency graph or the spill rules; the function packages exercise the evaluator indirectly through their fixtures.Packages/SheetIntelligence/EVALUATION.mdis a measurement of the on-device model on 20 fixed requests, with expected values computed independently in Swift from the generator of the synthetic data. It is an evaluation of suggestion quality, not a correctness proof; see ../architecture/intelligence.md.
5. Tolerances
| Package | Comparison | Tolerance |
|---|---|---|
| FunctionsNumeric | scipy / mpmath / numpy fixtures | relative 1e-12 (1e-11 for HYPGEOM.DIST, 2e-11 for BESSELJ / BESSELY, which rely on libm's jn / yn) |
| FunctionsNumeric | decimal rounding | exact (tolerance 0) |
| FunctionsNumeric | LibreOffice | 1e-9 to 1e-13 (LibreOffice writes 15 significant digits); its distribution functions at 1e-8 "because they are themselves less accurate" |
| FunctionsNumeric | text and logical values | exact |
| FunctionsLookup | all references | relative 1e-9; 1e-6 … 1e-8 for iterative results against Excel / LibreOffice, whose solvers stop earlier |
| SheetAnalytics | per routine | from rel 1e-14 (ln Γ) to rel 1e-7 (studentized-range quantile); amortization and Canadian mortgage schedules exact to the cent, every row; long-only optimiser weights abs 2e-6 (SLSQP's own accuracy) |
| SheetData | tables / pivots vs pandas | relative 1e-9 / 1e-8 |
| SheetFormat | rendered text | byte-identical, or one of the 13 documented deviations |
| SheetCharts | colour maps vs matplotlib | within 4 / 255 per channel |
Source: Packages/FunctionsNumeric/COMPATIBILITY.md ("Tolerances"), Packages/FunctionsLookup/COMPATIBILITY.md ("Comparison tolerance"), Packages/SheetAnalytics/VALIDATION.md (column "tolerance achieved"), Packages/SheetData/README.md, Packages/SheetFormat/COMPATIBILITY.md, Packages/SheetCharts/README.md.
"Tolerance achieved" in VALIDATION.md means the tolerance asserted by the passing tests; it is a bound
that holds on the tested grid, not a measured maximum error.
6. Where the references disagreed, and how it was arbitrated
| Disagreement | Cases | Arbitration |
|---|---|---|
scipy vs mpmath: poisson.pmf (4e-12 off for large means), t.isf / f.isf (up to 1e-8 off in the tails) |
3 places | mpmath at 40–400 digits; the fixtures use the mpmath values |
LibreOffice vs Excel's documentation, numeric functions (MROUND with mixed signs, POWER(0,0), POWER(-8,1/3), COMBINA(n<k), ATAN2(0,0), bit functions with non-integers, ISNUMBER(TRUE), wildcards and numeric criteria in COUNTIF / SUMIF, …) |
51 | overrides.py stores Excel's documented result instead of LibreOffice's |
| LibreOffice vs Excel's documentation, text and date functions (WEEKNUM at year end — 149 cases, booleans as text, error kinds, negative serials, FIXED rounding, UTF-16 length, …) | 234 | source tag excel: the Excel value with a note, the LibreOffice value kept in lo |
| LibreOffice vs Excel-computed data, financial functions (COUPDAYSNC around the end of February, time to the next coupon in PRICE / YIELD / DURATION, PRICEMAT / YIELDMAT, T-bills, …) | 4 472 | the case keeps the Python value only; the fixture records the LibreOffice value and the reason. "Each such reason was confirmed against the Excel data" |
LibreOffice vs Excel, number formats (serials below 61, seconds rounding, -0.00, ? padding, fraction approximation, A/P case, …) |
839 | 13 named deviation rules in ExcelDeviations.swift; ORIX Sheet follows Excel's documented display |
SciPy itself unreliable: Student t inverse for p < 1e-20 (overflows or stalls), f.isf below about 1e-8 |
— | closed forms (Cauchy, F(2,2)) and round trips sf(isf(q)) = q |
| NumPy vs SAS on quantile type 3 | — | no single reference: the method is not shipped |
| LibreOffice has no logical cell type (TRUE is the number 1) | — | every behaviour involving logical values inside ranges is covered by hand-written tests only |
| HOUR / MINUTE / SECOND rounding at half-second boundaries, DAYS truncation | — | Excel's observed behaviour as documented; LibreOffice differs, so hand-written tests only |
Source: Packages/FunctionsNumeric/COMPATIBILITY.md, Packages/FunctionsText/COMPATIBILITY.md, Packages/FunctionsLookup/COMPATIBILITY.md, Packages/SheetFormat/COMPATIBILITY.md (the 13 rows of its deviation table sum to 839), Packages/SheetAnalytics/VALIDATION.md.
The generators enforce this discipline mechanically: in FunctionsText the generator aborts when
LibreOffice and Python disagree on a date function; in FunctionsLookup "the generators fail when
LibreOffice and Python disagree without a recorded reason".
An arbitration in favour of "Excel's documented result" is only as good as the documentation. Items the
package documents flag as not verified include: LARGE / SMALL with a non-integer k, BITLSHIFT
overflow at 2^48, the position of errors and empty cells in SORT, coupon dates when maturity falls on the
29th or 30th and is not a month end, ENCODEURL keeping ~, four CONVERT unit factors (u, eV,
parsec, mmHg), and ROUND on values that differ from a tie only beyond the 15th digit. They are listed
function by function in ../compatibility/formulas.md.
7. Regenerating the fixtures
Requirements: /opt/homebrew/bin/python3 with the scientific packages and /opt/homebrew/bin/soffice
(LibreOffice) for the generators that recalculate workbooks. The generators are deterministic (fixed
seeds, fixed argument grids). Regeneration is a development task; the committed fixtures are what the
tests replay.
| Package | Command (from the package directory) |
|---|---|
| FunctionsNumeric | cd Tests/FunctionsNumericTests/Generators then run gen_libreoffice.py and gen_scipy.py (overrides.py holds the Excel overrides, coverage.py rebuilds the function table). FN_DUMP=<dir> swift test writes every mismatch of a regenerated corpus, one file per fixture |
| FunctionsText | python3 Tests/FunctionsTextTests/Generators/generate_function_fixtures.py (needs openpyxl, python-dateutil, soffice) |
| FunctionsLookup | cd Tests/FunctionsLookupTests/Generators then gen_lookup.py, gen_arrays.py, gen_financial.py, gen_securities.py; python3 -I gen_excel_reference.py <dir with the downloaded Excel data> (the script header gives the curl commands that download the public test data) |
| SheetFormat | python3 Tests/SheetFormatTests/Generators/generate_format_fixtures.py (needs openpyxl, soffice) |
| SheetAnalytics | `cd Tests/StatisticsTests/Generators && for f in gen_*.py; do [ $f = gen_common.py ] |
| SheetData | /opt/homebrew/bin/python3 Tests/SheetDataTests/Generators/generate_fixtures.py |
| SheetCharts | gen_fixtures.py and gen_colormaps.py in Tests/SheetChartsTests/Generators |
| SheetIO | gen_openpyxl_corpus.py OUTDIR, gen_raw_xlsx.py, then gen_libreoffice_corpus.sh FIXTURES_DIR in Tests/SheetIOTests/Generators |
| SheetTemplates | ORIX_TEMPLATES_INPUTS=Tests/SheetTemplatesTests/Generators/inputs.json swift test --filter exportGeneratorInputs then /opt/homebrew/bin/python3 Tests/SheetTemplatesTests/Generators/gen_expected.py |
Source: the "Regenerating the fixtures" sections of the package documents and the header comments of the generator scripts under Packages/*/Tests/*/Generators/.
After regenerating, review the diff of the fixture files before committing: a changed reference value is
either a change of the reference program's version or a mistake in the generator, and both deserve a
look. The only network access in the whole procedure is the download of the ExcelFinancialFunctions test
data for gen_excel_reference.py.
8. Attribution
Packages/FunctionsLookup/Tests/FunctionsLookupTests/Fixtures/excel_reference.json is derived from the
test data of ExcelFinancialFunctions (https://github.com/fsprojects/ExcelFinancialFunctions), licensed
under the Apache License, Version 2.0. The file states its origin and the changes made. The day-count
rules that Excel applies inside its bond functions were learned from that project's documentation of
Excel's behaviour and re-implemented in Swift (Packages/FunctionsLookup/COMPATIBILITY.md,
"Attribution").
9. What is not covered by any reference
- The formula parser, the dependency graph, spills and cycles: hand-written expectations only.
- The grid, the editor and the chrome: see ../development/testing.md.
- Rendering in Excel of files written by ORIX Sheet.
- Functions and options listed as "not shipped" in
Packages/SheetAnalytics/VALIDATION.md(for example two-way ANOVA on unbalanced designs, Fisher's exact test, seasonal ARIMA, GARCH standard errors, linear programming, the long-only maximum-Sharpe portfolio): they are absent from the public API because no validated reference was available or the method was out of scope. - Any behaviour marked "not verified" in the package documents (section 6).