SPB Git

Repository Public

reference-tests.md 27.0 KB 342 lines markdown

1# Reference tests23How the numerical and behavioural correctness of ORIX Sheet is established, what the references are,4how many cases each one contributes, the tolerances asserted, how to regenerate the fixtures — and where5the method stops.67Related documents: [numerical-precision.md](numerical-precision.md),8[financial-conventions.md](financial-conventions.md),9[../compatibility/formulas.md](../compatibility/formulas.md) (per-function validation source),10[../compatibility/excel.md](../compatibility/excel.md), [../development/testing.md](../development/testing.md)11(how to run the suites).1213## 1. Principle1415No result is accepted because the Swift code says so. Each computed value is compared with a number16produced by **a program that shares no code with ORIX Sheet**: another spreadsheet (LibreOffice), a17scientific library (SciPy, mpmath, NumPy, statsmodels, scikit-learn, pandas), exact decimal arithmetic18(Python `decimal`), a file-format library (openpyxl), values published as computed by Microsoft Excel,19or the worked examples of Microsoft's function reference typed by hand.2021The reference values are generated once by scripts kept next to the tests, committed as fixtures, and22replayed by `swift test` without Python, LibreOffice or a network connection.2324<!-- diagram: reference-tests-pipeline -->25```mermaid26flowchart LR27    G["Generator scripts<br/>Tests/*/Generators/*.py"] --> LO["LibreOffice headless<br/>recalculation"]28    G --> PY["Python references<br/>SciPy, mpmath, NumPy,<br/>statsmodels, pandas, decimal"]29    XL["Excel-computed public data<br/>(ExcelFinancialFunctions)"] --> CV["gen_excel_reference.py"]30    LO --> FX["Committed fixtures<br/>Tests/*/Fixtures/"]31    PY --> FX32    CV --> FX33    DOC["Microsoft documentation<br/>examples, typed by hand"] --> HW["Hand-written Swift tests"]34    FX --> ST["swift test<br/>fixture replay"]35    HW --> ST36    ST --> EN["ORIX Sheet engine<br/>and libraries"]37```38SVG fallback: ![Reference test pipeline](../assets/diagrams/reference-tests-pipeline.svg)3940## 2. The honest limits, first4142- **Microsoft Excel was never driven.** Excel is installed on the build machine43  (`docs/research/technology-evaluation.md`), but no test, generator or script launches it, opens a file44  in it or reads a value from it. Every package document repeats this: "Nothing in this document comes45  from running Excel" (`Packages/FunctionsNumeric/COMPATIBILITY.md`), "No result is validated against a46  live Excel" (`Packages/FunctionsText/COMPATIBILITY.md`), "Microsoft Excel itself was not run"47  (`Packages/SheetIO/COMPATIBILITY.md`).48- **The only values that come from Excel** are the public test data of the ExcelFinancialFunctions49  project (results computed by Excel for large argument matrices, Apache License 2.0), used for the50  financial functions and YEARFRAC. They cover valid inputs only (plus PDURATION / RRI failures); error51  kinds for invalid arguments rest on documentation and on LibreOffice returning *an* error.52- **Excel's behaviour elsewhere is taken from Microsoft's documentation.** Where the documentation is53  silent, the behaviour is either LibreOffice's (when the two programs are known to agree) or marked54  "not verified" in the package document and in [../compatibility/excel.md](../compatibility/excel.md).55- **LibreOffice is not Excel.** It disagrees with Excel's documented behaviour in several thousand of the56  generated cases ([section 6](#6-where-the-references-disagreed-and-how-it-was-arbitrated)). Those cases57  are not silently dropped: the fixture stores the Excel-documented value, a note and the LibreOffice58  value.59- **LibreOffice error codes are not compared.** Its XLSX export reports most errors as `#VALUE!` and60  logical values as 1/0, so fixtures check "is an error" (exact kind only for `#N/A` and `#DIV/0!` in61  `FunctionsLookup`); exact Excel error kinds are asserted by hand-written tests.62- **XLSX files written by ORIX Sheet were opened by openpyxl and LibreOffice only.** Nothing here is a63  claim that Excel opens them without a repair prompt.64- **A reference test proves agreement on the tested inputs**, not correctness everywhere. The grids are65  described in the package documents; outside them the code is unverified.6667## 3. Reference sources6869| Reference | Version stated in the package documents | Used for |70|---|---|---|71| LibreOffice Calc, headless recalculation of generated workbooks | 26.8 / 26.8.2 (en-US profile) | spreadsheet semantics of functions (coercion, blanks, criteria, ranges, arrays, text), number-format rendering, XLSX/ODS interoperability, template recalculation |72| Microsoft Excel-computed public data | ExcelFinancialFunctions test data (fsprojects, Copyright 2010-2013, Apache License 2.0) | financial functions and YEARFRAC |73| SciPy + mpmath | SciPy 1.17 (1.17.1), mpmath 1.3 at 40 digits (up to 400 for arbitration) | distributions, quantiles, special functions, Bessel, trigonometry, unit conversions, root finding, statistics |74| NumPy | 2.4 (2.4.4) | descriptive statistics, percentiles, histograms, linear algebra, portfolio measures |75| statsmodels | 0.14.6 | OLS / WLS / panel / logit regression blocks, LINEST statistics, prediction intervals, time series |76| scikit-learn | 1.6 | multivariate methods of `SheetAnalytics` (stated in `Packages/SheetAnalytics/VALIDATION.md`) |77| `arch` | 8 | GARCH |78| Python `decimal` | Python 3.14 standard library | ROUND family on the 15-digit decimal reading, MOD, QUOTIENT, amortization schedules to the cent (60-digit context) |79| Python `cmath`, `datetime`, `dateutil`, `re`, `str` | standard library + python-dateutil | complex functions, date functions, regular expressions and text functions LibreOffice lacks |80| pandas (+ numpy, pyarrow; **no DuckDB**) | pandas 3.0 (3.0.2) | data engine: aggregates, profiles, transforms, joins, pivot cubes; CSV round trips of `SheetIO` |81| openpyxl | 3.1.5 | independent reader of XLSX files written by ORIX Sheet, and generator of XLSX fixtures read by ORIX Sheet |82| matplotlib | version not stated | tick locator, box-plot statistics, colour maps, contour lengths (chart engine) |83| Microsoft function reference | worked examples typed by hand | documented results to the printed digits, documented error codes |8485Source: `Packages/FunctionsNumeric/COMPATIBILITY.md`, `Packages/FunctionsText/COMPATIBILITY.md`, `Packages/FunctionsLookup/COMPATIBILITY.md`, `Packages/SheetAnalytics/VALIDATION.md`, `Packages/SheetFormat/COMPATIBILITY.md`, `Packages/SheetIO/COMPATIBILITY.md`, `Packages/SheetData/README.md`, `Packages/SheetCharts/README.md`, `docs/research/technology-evaluation.md` (installed versions).8687## 4. Case counts per package8889The counts in the "Stated" column are copied from the package documents. The "Counted" column was90obtained on 2026-11-21 by loading the committed fixture files with Python and counting their cases; it is91there to show that the documents and the files agree.9293| Package | Fixture(s) | Stated | Counted in the files |94|---|---|---:|---:|95| FunctionsNumeric | 24 JSON files in `Tests/FunctionsNumericTests/Fixtures/` | 15 378 | 15 378 |96| FunctionsText | `function_references.json` | 7 951 | 7 951 |97| FunctionsLookup | `excel_reference.json`, `financial.json`, `securities.json`, `lookup_lo.json`, `arrays.json` | 86 965 formula cases + 1 397 YEARFRAC rows | 86 965 + 1 397 |98| SheetFormat | `libreoffice_formats.json` | 5 085 | 5 085 |99100Source: package documents named in section 3; counts measured with a Python script over `Packages/*/Tests/*/Fixtures/*.json` on 2026-11-21.101102### FunctionsNumeric (296 functions)103104| Source | Cases | What it vouches for |105|---|---:|---|106| LibreOffice 26.8 (headless recalculation of generated `.xlsx` workbooks, en-US locale) | 4 687 | spreadsheet semantics: coercion, blanks, criteria, ranges, array results, text results |107| scipy 1.17 + mpmath 1.3 (40 digits) | 5 440 | distributions, quantiles, special functions, Bessel, trigonometry, unit conversions (`scipy.constants`) |108| Python `decimal` | 4 082 | ROUND / ROUNDUP / ROUNDDOWN / TRUNC on the 15-digit decimal reading, MOD, QUOTIENT |109| numpy + scipy.stats | 606 | descriptive statistics, percentiles, ranks, correlation, t / F / z / χ² tests |110| statsmodels OLS + numpy.linalg + mpmath | 143 | LINEST / LOGEST statistics blocks, TREND, GROWTH, MINVERSE, MDETERM, MMULT |111| Python `cmath` | 420 | complex-number functions |112| **Reference corpus (JSON fixtures)** | **15 378** | |113| Microsoft documentation examples (hand-written) | ≈ 370 | `ExcelDocumentationTests.swift`: the examples of Excel's help pages, to the printed digits |114| Hand-written edge cases | ≈ 500 | `EdgeCaseTests.swift`: 201 Excel error codes, logical values, arrays, blanks, whole columns, hidden rows |115116Source: `Packages/FunctionsNumeric/COMPATIBILITY.md` ("How the functions are validated"). The per-source totals were recounted from the `source` field of the 24 fixture files on 2026-11-21 and match (libreoffice 4 687, scipy 5 440, decimal 4 082, numpy 606, statsmodels 143, cmath 420).117118The per-function breakdown (LibreOffice cases, independent cases, hand-written cases for each of the 296119functions) is the "Function table" of `Packages/FunctionsNumeric/COMPATIBILITY.md`, reproduced in120[../compatibility/formulas.md](../compatibility/formulas.md).121122### FunctionsText (64 functions: 41 text, 23 date and time)123124| Source tag | Cases | Meaning |125|---|---:|---|126| `lo` | 3 565 | formula recalculated by headless LibreOffice 26.8 (en-US profile) |127| `lo+py` | 1 790 | LibreOffice result that also equals an independent Python reference (`datetime`, `dateutil`; the generator aborts on any disagreement): YEAR, MONTH, DAY, HOUR, MINUTE, SECOND, WEEKDAY, WEEKNUM, ISOWEEKNUM, EDATE, EOMONTH, DAYS, NETWORKDAYS, WORKDAY |128| `py` | 2 362 | Python reference (`re`, `str`) for the functions LibreOffice does not implement: VALUETOTEXT, ARRAYTOTEXT, REGEXTEST, REGEXEXTRACT, REGEXREPLACE |129| `excel` | 234 | LibreOffice deviates from Excel's documented behaviour; the Excel value is stored with a note, the LibreOffice value is kept in `lo` |130| **Total** | **7 951** | |131132Source: `Packages/FunctionsText/COMPATIBILITY.md` ("Validation"); the four totals were recounted from the `src` field of `function_references.json` on 2026-11-21 and match; the file covers 64 distinct function names.133134About 310 hand-written expectations add Microsoft's documented examples, array lifting, ranges, blanks,135errors, UTF-16 positions, the 1900 leap bug, the 1904 system and the fr-CA locale. The reference test136also fails when a registered function has no reference case.137138### FunctionsLookup (81 functions: 10 lookup and reference, 22 dynamic-array, 49 financial)139140| Tag | Source | Cases |141|---|---|---|142| **XL** | Values computed by Microsoft Excel: test data of ExcelFinancialFunctions, converted by `Generators/gen_excel_reference.py` into `Fixtures/excel_reference.json` (dates → serials, enumerations → Excel codes; nothing else changed, nothing thinned) | 68 075 + 1 397 YEARFRAC |143| **LO** | LibreOffice Calc 26.8.2 headless recalculation of generated workbooks (`Generators/lo_harness.py`) | 14 223 |144| **PY** | Independent Python computations (`Generators/pyref.py`, `gen_arrays.py`): closed forms, period-by-period simulations, explicit cash-flow discounting, `scipy.optimize.brentq` | 17 874 (13 207 of them also confirmed by LO) |145| **DOC** | Worked examples of Microsoft's function reference, typed by hand in `FinancialTests.swift` | 90 |146| **HW** | Hand-written Excel-semantics tests (`LookupTests`, `DynamicArrayTests`, `FinancialTests`, `PublicAPITests`) | ≈ 500 assertions |147148Source: `Packages/FunctionsLookup/COMPATIBILITY.md` ("Reference sources"). File sizes counted on 2026-11-21: `excel_reference.json` 68 075 cases + 1 397 YEARFRAC rows, `securities.json` 13 153, `financial.json` 4 591, `lookup_lo.json` 847, `arrays.json` 299 (sum of the four formula files and the Excel file: 86 965).149150The number of Excel-computed cases per financial function (for example PRICE 10 982, DURATION 5 492,151IPMT 4 552, VDB 2 546, RATE 8, IRR 11) is in the "Financial" table of152`Packages/FunctionsLookup/COMPATIBILITY.md`. Three gaps are stated there and matter:153154- **XNPV has no Excel data set** (LibreOffice + Python + documentation example).155- **YIELD**: the Excel set covers negative yields (441 cases); positive yields rest on PRICE being156  inverted (10 982 Excel cases for PRICE), on LibreOffice + Python and on the documentation example.157- **ACCRINT**: the Excel data covers settlement ≤ `first_interest` with `calc_method` TRUE only.158159### SheetFormat (number formats)160161- 5 085 `=TEXT(value, format)` results computed by headless LibreOffice 26.8 (70 numeric formats × 44162  values, 61 date/time formats × 32 serials, text cases). **4 246 are byte-identical; 839 differ in one163  of 13 documented ways**, each checked by a narrow rule in `ExcelDeviations.swift` so that any other164  difference fails the test.165- About 340 hand-written expectations (`EdgeTests.swift`).166- A fuzz test renders 20 000 random format codes against 12 values in two locales (no crash, bounded167  General width).168169Source: `Packages/SheetFormat/COMPATIBILITY.md` ("Validation").170171### SheetAnalytics (statistics, finance models, what-if, audit)172173`Packages/SheetAnalytics/VALIDATION.md` lists every public routine with its reference, the tolerance174asserted and its status. It states 305 tests and about 2 600 `#expect` sites, many inside loops over175fixture cases; the document does not give a number of reference cases, so none is quoted here.176177| Section of VALIDATION.md | Rows in the table | References named |178|---|---:|---|179| Statistics — core, distributions, descriptive statistics, inference | 51 | `scipy.special`, `scipy.stats`, closed forms and round trips where SciPy is unreliable, pure-Python xoshiro256** reference |180| Statistics — regression and econometrics | 27 | statsmodels (OLS, WLS, robust errors, panel, `Logit`), numpy |181| Statistics — time series, multivariate, least squares | 29 | statsmodels, scikit-learn, `arch`, SciPy |182| Statistics — resampling; WhatIf target | 25 | `scipy.stats.bootstrap` with the same index matrix, hand-derived exact expectations |183| FinanceModels | 30 | independent Python formulas, Python `decimal` schedules, `scipy.optimize.brentq`, `numpy.roots`, SLSQP, Excel documentation examples |184| Audit — Model Auditor and Formula Intelligence | 12 | hand-derived exact expectations on workbooks built in the tests |185| Audit — conditional formatting, validation, tables | 20 | hand-derived exact expectations |186187Source: `Packages/SheetAnalytics/VALIDATION.md`; row counts obtained by counting the table lines of each section (header and separator lines excluded) on 2026-11-21. Rows include the entries marked "not shipped".188189Fixtures: `Tests/StatisticsTests/Fixtures/` (`core.json`, `descriptive_inference.json`,190`regression.json`, `timeseries.json`, `multivariate.json`, `resampling.json`, `tukey.json`) and191`Tests/FinanceModelsTests/Fixtures/finance.json`.192193### SheetData (data engine)194195`Tests/SheetDataTests/Generators/generate_fixtures.py` (pandas, numpy, pyarrow — no DuckDB) generates196seeded synthetic files (CSV, Parquet, JSON Lines, SQLite, Arrow, CSV dialects, hostile column names) and197`expected.json`: aggregates, profiles, the expected table of every transform scenario, joins of each198kind, and pivot cubes computed with `groupby` at every (row prefix, column prefix) level, show-as199variants, sort orders and drill-down row sets. The Swift tests compare the engine's output with those200references cell by cell.201202Source: `Packages/SheetData/README.md` ("Tests").203204### SheetIO (file formats)205206There is no numeric corpus: the references are other programs reading what ORIX Sheet writes and207writing what ORIX Sheet reads. `Tests/SheetIOTests/Fixtures/` holds files produced by openpyxl208(`features_openpyxl.xlsx`, `charts_openpyxl.xlsx`, `dates1904_openpyxl.xlsx`), by LibreOffice209(`features_lo.xlsx`, `features_lo.ods`), a hand-built raw XLSX (`raw_features.xlsx`) and a macro workbook210(`macro.xlsm`). `InteropTests.swift` runs openpyxl, pandas and LibreOffice on files written by the211tests. The feature-by-feature "Verified by" column is in `Packages/SheetIO/COMPATIBILITY.md` and in212[../compatibility/formats.md](../compatibility/formats.md).213214### SheetCharts (chart numerics)215216`Tests/SheetChartsTests/Fixtures/reference.json` and `colormaps.json` are generated by `gen_fixtures.py`217and `gen_colormaps.py` (numpy 2.4.4, scipy 1.17.1, matplotlib): tick positions218(`matplotlib.ticker.MaxNLocator`), histogram bins (`numpy.histogram_bin_edges`), quantile types219(`numpy.quantile`), box-plot statistics (`matplotlib.cbook.boxplot_stats`), kernel density220(`scipy.stats.gaussian_kde`), trendlines, stacking, contour lengths and colour maps (within 4/255 of221matplotlib). The treemap is covered by property tests, not by a reference; the 3-D surface has no222numerical reference.223224Source: `Packages/SheetCharts/README.md` ("Statistical conventions").225226### SheetTemplates227228`Tests/SheetTemplatesTests/Generators/gen_expected.py` recomputes the key outputs of every229template with numpy / scipy and closed forms ("never with a spreadsheet engine") into230`Fixtures/expected.json`, which `NumericTests.swift` compares with the calculated workbooks; and231`LibreOfficeTests.swift` exports every template with poisoned cached values, lets headless LibreOffice232recalculate it and compares every formula cell and spilled cell at a relative tolerance of 1e-6. The233last run in `build/test-logs/SheetTemplates.log` compared 18,714 cells in 30 workbooks (15 templates ×2342 languages) with 0 mismatches. One waiver is recorded in the test: 22 cells of the French235sales-tracking template (`Sommaire!I10:K17`, a `SORT` of accented product names) are not compared. The236waiver was written when ORIX Sheet ordered text by code point; text ordering now uses a fixed237collation ([numerical-precision.md](numerical-precision.md)) but the waiver is still in the test, so238those cells remain unchecked against LibreOffice. See [../templates.md](../templates.md).239240### Core engine (SheetCore, SheetEngine) and intelligence241242- `Packages/SheetCore/Tests/SheetCoreTests/CoreTests.swift` (addresses, parser, semantics, recalculation,243  editing, input) and `Packages/SheetEngine/Tests/SheetEngineTests/EngineTests.swift` are **hand-written244  expectations**. There is no external reference corpus for the parser, the dependency graph or the245  spill rules; the function packages exercise the evaluator indirectly through their fixtures.246- `Packages/SheetIntelligence/EVALUATION.md` is a measurement of the on-device model on 20 fixed247  requests, with expected values computed independently in Swift from the generator of the synthetic248  data. It is an evaluation of suggestion quality, not a correctness proof; see249  [../architecture/intelligence.md](../architecture/intelligence.md).250251## 5. Tolerances252253| Package | Comparison | Tolerance |254|---|---|---|255| FunctionsNumeric | scipy / mpmath / numpy fixtures | relative 1e-12 (1e-11 for HYPGEOM.DIST, 2e-11 for BESSELJ / BESSELY, which rely on libm's `jn` / `yn`) |256| FunctionsNumeric | decimal rounding | exact (tolerance 0) |257| FunctionsNumeric | LibreOffice | 1e-9 to 1e-13 (LibreOffice writes 15 significant digits); its distribution functions at 1e-8 "because they are themselves less accurate" |258| FunctionsNumeric | text and logical values | exact |259| FunctionsLookup | all references | relative 1e-9; 1e-6 … 1e-8 for iterative results against Excel / LibreOffice, whose solvers stop earlier |260| SheetAnalytics | per routine | from rel 1e-14 (ln Γ) to rel 1e-7 (studentized-range quantile); amortization and Canadian mortgage schedules **exact to the cent, every row**; long-only optimiser weights abs 2e-6 (SLSQP's own accuracy) |261| SheetData | tables / pivots vs pandas | relative 1e-9 / 1e-8 |262| SheetFormat | rendered text | byte-identical, or one of the 13 documented deviations |263| SheetCharts | colour maps vs matplotlib | within 4 / 255 per channel |264265Source: `Packages/FunctionsNumeric/COMPATIBILITY.md` ("Tolerances"), `Packages/FunctionsLookup/COMPATIBILITY.md` ("Comparison tolerance"), `Packages/SheetAnalytics/VALIDATION.md` (column "tolerance achieved"), `Packages/SheetData/README.md`, `Packages/SheetFormat/COMPATIBILITY.md`, `Packages/SheetCharts/README.md`.266267"Tolerance achieved" in `VALIDATION.md` means the tolerance asserted by the passing tests; it is a bound268that holds on the tested grid, not a measured maximum error.269270## 6. Where the references disagreed, and how it was arbitrated271272| Disagreement | Cases | Arbitration |273|---|---:|---|274| scipy vs mpmath: `poisson.pmf` (4e-12 off for large means), `t.isf` / `f.isf` (up to 1e-8 off in the tails) | 3 places | mpmath at 40–400 digits; the fixtures use the mpmath values |275| LibreOffice vs Excel's documentation, numeric functions (MROUND with mixed signs, `POWER(0,0)`, `POWER(-8,1/3)`, `COMBINA(n<k)`, `ATAN2(0,0)`, bit functions with non-integers, `ISNUMBER(TRUE)`, wildcards and numeric criteria in COUNTIF / SUMIF, …) | 51 | `overrides.py` stores Excel's documented result instead of LibreOffice's |276| LibreOffice vs Excel's documentation, text and date functions (WEEKNUM at year end — 149 cases, booleans as text, error kinds, negative serials, FIXED rounding, UTF-16 length, …) | 234 | source tag `excel`: the Excel value with a note, the LibreOffice value kept in `lo` |277| LibreOffice vs Excel-computed data, financial functions (COUPDAYSNC around the end of February, time to the next coupon in PRICE / YIELD / DURATION, PRICEMAT / YIELDMAT, T-bills, …) | 4 472 | the case keeps the Python value only; the fixture records the LibreOffice value and the reason. "Each such reason was confirmed against the Excel data" |278| LibreOffice vs Excel, number formats (serials below 61, seconds rounding, `-0.00`, `?` padding, fraction approximation, `A/P` case, …) | 839 | 13 named deviation rules in `ExcelDeviations.swift`; ORIX Sheet follows Excel's documented display |279| SciPy itself unreliable: Student t inverse for p < 1e-20 (overflows or stalls), `f.isf` below about 1e-8 | — | closed forms (Cauchy, F(2,2)) and round trips `sf(isf(q)) = q` |280| NumPy vs SAS on quantile type 3 | — | no single reference: the method is **not shipped** |281| LibreOffice has no logical cell type (TRUE is the number 1) | — | every behaviour involving logical values inside ranges is covered by hand-written tests only |282| HOUR / MINUTE / SECOND rounding at half-second boundaries, DAYS truncation | — | Excel's observed behaviour as documented; LibreOffice differs, so hand-written tests only |283284Source: `Packages/FunctionsNumeric/COMPATIBILITY.md`, `Packages/FunctionsText/COMPATIBILITY.md`, `Packages/FunctionsLookup/COMPATIBILITY.md`, `Packages/SheetFormat/COMPATIBILITY.md` (the 13 rows of its deviation table sum to 839), `Packages/SheetAnalytics/VALIDATION.md`.285286The generators enforce this discipline mechanically: in `FunctionsText` the generator aborts when287LibreOffice and Python disagree on a date function; in `FunctionsLookup` "the generators fail when288LibreOffice and Python disagree without a recorded reason".289290An arbitration in favour of "Excel's documented result" is only as good as the documentation. Items the291package documents flag as **not verified** include: LARGE / SMALL with a non-integer `k`, BITLSHIFT292overflow at 2^48, the position of errors and empty cells in SORT, coupon dates when maturity falls on the29329th or 30th and is not a month end, `ENCODEURL` keeping `~`, four CONVERT unit factors (`u`, `eV`,294`parsec`, `mmHg`), and ROUND on values that differ from a tie only beyond the 15th digit. They are listed295function by function in [../compatibility/formulas.md](../compatibility/formulas.md).296297## 7. Regenerating the fixtures298299Requirements: `/opt/homebrew/bin/python3` with the scientific packages and `/opt/homebrew/bin/soffice`300(LibreOffice) for the generators that recalculate workbooks. The generators are deterministic (fixed301seeds, fixed argument grids). Regeneration is a development task; the committed fixtures are what the302tests replay.303304| Package | Command (from the package directory) |305|---|---|306| FunctionsNumeric | `cd Tests/FunctionsNumericTests/Generators` then run `gen_libreoffice.py` and `gen_scipy.py` (`overrides.py` holds the Excel overrides, `coverage.py` rebuilds the function table). `FN_DUMP=<dir> swift test` writes every mismatch of a regenerated corpus, one file per fixture |307| FunctionsText | `python3 Tests/FunctionsTextTests/Generators/generate_function_fixtures.py` (needs openpyxl, python-dateutil, soffice) |308| FunctionsLookup | `cd Tests/FunctionsLookupTests/Generators` then `gen_lookup.py`, `gen_arrays.py`, `gen_financial.py`, `gen_securities.py`; `python3 -I gen_excel_reference.py <dir with the downloaded Excel data>` (the script header gives the `curl` commands that download the public test data) |309| SheetFormat | `python3 Tests/SheetFormatTests/Generators/generate_format_fixtures.py` (needs openpyxl, soffice) |310| SheetAnalytics | `cd Tests/StatisticsTests/Generators && for f in gen_*.py; do [ $f = gen_common.py ] || /opt/homebrew/bin/python3 $f; done` (same in `Tests/FinanceModelsTests/Generators`) |311| SheetData | `/opt/homebrew/bin/python3 Tests/SheetDataTests/Generators/generate_fixtures.py` |312| SheetCharts | `gen_fixtures.py` and `gen_colormaps.py` in `Tests/SheetChartsTests/Generators` |313| SheetIO | `gen_openpyxl_corpus.py OUTDIR`, `gen_raw_xlsx.py`, then `gen_libreoffice_corpus.sh FIXTURES_DIR` in `Tests/SheetIOTests/Generators` |314| SheetTemplates | `ORIX_TEMPLATES_INPUTS=Tests/SheetTemplatesTests/Generators/inputs.json swift test --filter exportGeneratorInputs` then `/opt/homebrew/bin/python3 Tests/SheetTemplatesTests/Generators/gen_expected.py` |315316Source: the "Regenerating the fixtures" sections of the package documents and the header comments of the generator scripts under `Packages/*/Tests/*/Generators/`.317318After regenerating, review the diff of the fixture files before committing: a changed reference value is319either a change of the reference program's version or a mistake in the generator, and both deserve a320look. The only network access in the whole procedure is the download of the ExcelFinancialFunctions test321data for `gen_excel_reference.py`.322323## 8. Attribution324325`Packages/FunctionsLookup/Tests/FunctionsLookupTests/Fixtures/excel_reference.json` is derived from the326test data of ExcelFinancialFunctions (https://github.com/fsprojects/ExcelFinancialFunctions), licensed327under the Apache License, Version 2.0. The file states its origin and the changes made. The day-count328rules that Excel applies inside its bond functions were learned from that project's documentation of329Excel's behaviour and re-implemented in Swift (`Packages/FunctionsLookup/COMPATIBILITY.md`,330"Attribution").331332## 9. What is not covered by any reference333334- The formula parser, the dependency graph, spills and cycles: hand-written expectations only.335- The grid, the editor and the chrome: see [../development/testing.md](../development/testing.md).336- Rendering in Excel of files written by ORIX Sheet.337- Functions and options listed as "not shipped" in `Packages/SheetAnalytics/VALIDATION.md` (for example338  two-way ANOVA on unbalanced designs, Fisher's exact test, seasonal ARIMA, GARCH standard errors,339  linear programming, the long-only maximum-Sharpe portfolio): they are absent from the public API340  because no validated reference was available or the method was out of scope.341- Any behaviour marked "not verified" in the package documents (section 6).342
Files main
Menu

Theme