Scientific data engine (scientific) — compatibility
Package Packages/EngineScientific, module EngineScientific, factory makeEngines(). HDF5 and NetCDF-4 go
through the bundled HDF5 2.2.0 C library (EmbeddedEngines/hdf5, BSD 3-Clause, static, see its NOTICE.md) behind
a C shim (Sources/CHDF5Shim). NetCDF classic (CDF-1, CDF-2, CDF-5) is read and written by the engine's own Swift
code. CSV, JSON and Parquet output are written by the engine itself (no DuckDB dependency).
State on 2026-11-21 (macOS 27.0, arm64): 10 declared paths, 48 tests, all passing
(swift test: Test run with 48 tests in 12 suites passed; one of the 48 is a development aid that only runs when
an environment variable is set). The full suite was run 10 times in a row with identical results.
Why no netCDF library
NetCDF-4 files are HDF5 files that follow conventions; classic files are a small, fully specified binary format. Building netcdf-c would add a second large C parser in front of untrusted files (plus DAP, Zarr and plug-in code to switch off) for no capability the engine lacks. Instead:
- classic: own reader and writer from the format specification, bounds-checked, tested against files
assembled byte by byte in the tests (
NetCDFInspectTests.classicFromIndependentBytes, three variants) — and the writer must reproduce those bytes exactly (NetCDFConversionTests.classicWriterReproducesIndependentBytes); - NetCDF-4: read and written through HDF5 directly, honouring the NetCDF-4 conventions — dimension scales,
dimension-only placeholder datasets (
This is a netCDF dimension but not a netCDF variable…),_Netcdf4Dimid,_NCProperties, the_nc4_non_coord_prefix, creation-order tracking, unlimited dimensions,_FillValue. Tested against a NetCDF-4 file built with raw dimension-scale calls (NetCDFInspectTests.netCDF4BuiltWithRawCalls) and by inspecting the engine's output with the C library (HDF5ToNetCDFTests.everythingToNetCDF4).
During development the engine's outputs were opened with netCDF4-python 1.7.5 (netcdf-c 4.10.1), h5py 3.16, pyarrow 26.0.1 and DuckDB 2.0.1 in a scratch environment: netcdf-c reads the hand-assembled classic fixtures and every NetCDF file the engine writes (dimensions, unlimited flags, attributes, filters, values); h5py reads the HDF5 outputs; pyarrow and DuckDB read the Parquet files with the right types, nulls and key/value metadata. That check found two defects that are now handled (a dataset reachable through a hard link in another group, and compound types with text members, both of which netcdf-c cannot read). None of those tools is used by the tests or shipped.
Declared paths
| Source | Destination | Operation | Status | Fidelity | Preserves | Tests |
|---|---|---|---|---|---|---|
| hdf5 | csv | — | supported | partial | — | FlattenCSVTests (9 tests) |
| hdf5 | parquet | — | supported | partial | metadata | FlattenParquetTests.typesNullsAndMetadata, severalRowGroups |
| hdf5 | json | — | supported | partial | metadata | FlattenJSONTests.records, nestedArrays |
| netcdf | csv | — | supported | partial | — | FlattenCSVTests.netCDFSources (classic and NetCDF-4) |
| netcdf | parquet | — | supported | partial | metadata | EngineSurfaceTests.netCDFToParquet (classic and NetCDF-4) |
| netcdf | json | — | supported | partial | metadata | FlattenJSONTests.nestedArrays (classic) |
| hdf5 | netcdf | — | supported | structural | metadata, attributes, dimensions | HDF5ToNetCDFTests.everythingToNetCDF4, classicVariantsFromHDF5, classicRefusals |
| netcdf | hdf5 | — | supported | structural | metadata, attributes, dimensions | NetCDFConversionTests.classicToNetCDF4AndBack, netCDF4ToHDF5AndRecompression |
| netcdf | netcdf | netcdf.reformat (Change NetCDF Variant) |
supported | structural | metadata, attributes, dimensions | classicWriterReproducesIndependentBytes, classicToNetCDF4AndBack, netCDF4ToHDF5AndRecompression, severalUnlimitedDimensionsToClassic |
| hdf5 | hdf5 | hdf5.repack (Repack / Recompress HDF5) |
supported | structural | metadata, attributes, dimensions | RepackTests.repackKeepsEverythingItCan, cancellationAndProgress |
netcdf covers the three classic variants and NetCDF-4; the container is recognised from the content. Limitations
shown to the user: tabular outputs are a flattened view of the selected variables, not the file; CSV has no place
for attributes; JSON turns NaN / infinity into null; references, bitfields, half floats and the HDF5 time class
have no NetCDF equivalent; classic output is refused for groups, strings and user-defined types; external links,
external raw storage and virtual datasets are never followed; szip and third-party filters are not available.
Inspection
inspect returns the container, the numbers of groups and datasets, the dimensions (name, length, unlimited),
global attributes with values, links that are not followed, and one line per dataset: type, shape with dimension
names, chunk shape, filters with the deflate level, fill value, coordinate-variable flag, CF hints exactly as read
(units, standard_name, calendar, packing) and the attribute count (first 300 datasets, 5000 objects walked).
The richer API for the app's dataset browser is SciFile:
| Call | Result |
|---|---|
SciFile.open(url) (async; runs the isolated check when the helper is bundled) / SciFile(url:) |
the file, whatever its container |
children(of:offset:limit:) |
lazy tree, paged; creation order where the file tracks it |
node(at:) |
kind (group, dataset, named type, external link with its target, dangling link), SciDatasetInfo, attribute count, soft-link target |
SciDatasetInfo |
type (integers of every width and signedness, floats, fixed / variable strings, char, compound, enum, array, vlen, reference, opaque, bitfield, time), dimensions (name, length, unlimited, coordinate variable, defining scale), layout, chunk shape, filters (with availability), fill value, stored bytes, why it cannot be read if so |
attributes(of:offset:limit:maxElements:), attribute(_:of:offset:maxElements:) |
paged attributes, each with a bounded number of values (attributes over 16 MiB are reported, not loaded) |
conventions(of:) |
units, standard_name, long_name, calendar, axis, scale_factor, add_offset, _FillValue, missing_value |
preview(_:start:count:stride:maxElements:) |
a bounded rectangular slice of any dataset, as text; compound members as columns |
tree(maxNodes:) |
bounded depth-first listing |
Dimension-scale bookkeeping (CLASS, NAME on scales, DIMENSION_LIST, REFERENCE_LIST) and NetCDF-4
internals (_NCProperties, _Netcdf4Dimid, _Netcdf4Coordinates, dimension placeholders) are presented as
dimensions, not as attributes or datasets. Tests: HDF5InspectTests (8), NetCDFInspectTests (2).
Flattening to tables (CSV, Parquet, JSON)
Flattening is explicit. Nothing is converted until the engine knows which variables, which index ranges and which layout; the exact row count is computed before any data is read.
- Variables (
variables): dataset paths separated by commas. Left empty, the conversion proceeds only when the file holds exactly one candidate (a dataset of a tabular type that is not a coordinate variable or a soft link); otherwise it stops with aninvalidOptionerror listing the candidates with type and shape —preflightreturns the same text as a warning (sci-selection-needed) for the app to present. Several variables must share the same dimensions. - Index ranges (
subset):dimension=start:stop:stride, comma-separated;stopexclusive, 0-based, parts optional (time=10:,lat=::2,level=3). Unnamed dimensions aredim0,dim1…, or their position. - Layout (
layout):
| Layout | Rows | Columns |
|---|---|---|
long |
one per selected cell, C order (last dimension fastest) | one per dimension — the coordinate variable's values when there is one, else the index — then one per variable (one per member for a compound) |
wide |
one per index of the first dimension | first dimension, then one column per index of the second, named dim=label; exactly one 2-D variable; ≤ 10 000 columns |
table |
one per element | one per variable or compound member, no dimension column; 1-D variables and scalars |
auto (default) |
table for a scalar or a single dimension without coordinate values (or when the coordinate variable is itself selected); long otherwise |
- Fill values (
fill):keep(default) ornull— cells equal to_FillValue/missing_valuebecome null (integers compared as integers, float32 at float32 precision, NaN fills recognised). The HDF5 fill-value property alone is not treated as "missing". - Packed values (
packed):raw(default) leaves the stored numbers;unpackwritesvalue × scale_factor + add_offsetas float64. Withraw, a caution warning says the table does not hold the physical quantity. - Time (
time.decode, off by default):<unit> since <date>numbers become ISO 8601 UTC text, for thestandard,gregorianandproleptic_gregoriancalendars only; units from microseconds to days. Other calendars (noleap,360_day,julian…), month / year units and dates before 15 October 1582 in the mixed calendar stay numbers, with a warning naming the calendar and the units. - Row limit (
rows.limit, default 10 000 000): the conversion is refused above it; raising the option is the confirmation. - Text stored as characters along a last dimension (NetCDF
char) is folded into one string per row. Enumerations are written as their names, array-typed elements as[a,b,c]text.
Example (/chunked/grid, int16 20 × 30 with scales y, x; options subset = y=0:2, x=0:3):
long wide (layout=wide, x=0:4, y=0:3) long, fill=null, packed=unpack
y,x,grid y,x=10.0,x=10.5,x=11.0,x=11.5 y,x,grid
0,10.0,-999 0,-999,1,2,3 0,10.0,
0,10.5,1 2,100,101,102,103 0,10.5,273.15999999999997
0,11.0,2 4,200,201,202,203 0,11.0,273.16999999999996
2,10.0,100The transformation is recorded, not only applied: in the facts (sci.transformation, sci.layout,
sci.rows, sci.nulls…), in a sci-flattened warning before and during conversion, and in the file:
| Target | Where |
|---|---|
| Parquet | key/value metadata orix_convert.transformation, orix_convert.attributes, orix_convert.source (JSON text) |
| JSON | the metadata object; records shape {"metadata", "columns", "rows":[{…}]} or nested shape {"metadata", "variable", "dimensions", "shape", "data": nested arrays} (one non-compound variable) |
| CSV | nowhere by default (caution warning counting the attributes); with metadata.sidecar the result is a folder holding name.csv and name.metadata.json |
The transformation record lists the source container, layout, row order, row count, each variable (path, type, dimensions, shape, units, calendar, what was done with packing, fill values and time) and each dimension (length, start, count, stride, whether labels are indices or the values of which variable).
Parquet files: one row group per block of at most 1 048 576 rows; PLAIN encoding; exact integer widths and
signedness (INT_8…UINT_64 annotations), FLOAT, DOUBLE, UTF-8 strings; nullable columns only where nulls can
occur; uncompressed by default, gzip pages on request. No dictionary encoding, statistics or page index.
Structure-preserving conversions
| From → to | Carried | Refused or reported |
|---|---|---|
| HDF5 → NetCDF-4 | groups, datasets with their exact types (incl. enum, opaque, variable-length, compound of numbers), attributes, chunk shapes, deflate / shuffle / Fletcher-32, fill values, unlimited dimensions, dimension scales as named dimensions | left out with one dataLoss warning each: datasets holding references, bitfields, half or extended floats, the time class, array-typed datasets, compounds with text / enum / vlen / opaque members. Dimensions without a scale → phony_dim_N. A scale outside the variable's group hierarchy → generated dimension + warning. Hard-link aliases and soft links are not NetCDF notions (warning) |
| HDF5 → NetCDF classic | root-level numeric and character datasets, attributes | refused: any dataset inside a group, strings, compound / enum / vlen / opaque / array types. Unsigned and 64-bit integers need CDF-5 (chosen automatically, or refused if an older variant is forced). Only one unlimited dimension, leading its variables; others become fixed (warning). String-array attributes joined line by line (warning). Names adapted to NetCDF rules (warning) |
| NetCDF classic → NetCDF-4 / HDF5 | every dimension (also unused ones), variable, attribute, the record dimension as unlimited, _FillValue also as the HDF5 fill value; optional deflate + shuffle |
nothing |
| NetCDF-4 → HDF5 | same as HDF5 → HDF5; the file keeps its NetCDF-4 dimension machinery and stays readable as NetCDF-4 | same as repack |
| NetCDF-4 → classic | as HDF5 → classic | as HDF5 → classic; chunking and compression do not exist (info warning) |
| classic → classic (variant change) | byte-identical when the variant is unchanged; otherwise same content | content that does not fit the chosen variant is refused, never truncated |
| HDF5 → HDF5 (repack) | everything above plus HDF5-only types (compound with text, arrays, half floats, bitfields), hard links and soft links recreated, named types | datasets holding references are left out; external links and dangling soft links are not recreated; filters other than deflate / shuffle / Fletcher-32 are not re-applied (values kept) |
compression: preserve (default), none, deflate with compression.level 1–9 and compression.shuffle.
netcdf.format: auto (NetCDF-4 from HDF5 or classic; the smallest fitting classic variant from NetCDF-4),
netcdf4, classic, 64bit-offset, cdf5, classic-auto.
scale_factor / add_offset and fill values are attributes like any other in these conversions: copied,
never applied.
Large data
Data moves in rectangular blocks of at most 1 048 576 elements (or 16 MiB), aligned to the chunk grid when the
source is chunked; progress is the measured share of elements written (reported every 0.5 %); cancellation is
checked between blocks. Measured on an M5 Max (48 GB), release build, EngineScientificBench (each line a
separate process; peak resident memory from getrusage; validation included in the peak):
| Run | Source | Output | Convert | Throughput | Peak RSS |
|---|---|---|---|---|---|
| classic (CDF-5) → HDF5, 300 M float64 | 2.24 GiB | 2.24 GiB | 1.1 s | 280 M values/s | 42 MiB |
| HDF5 → Parquet, 300 M rows | 2.24 GiB | 2.24 GiB (287 row groups) | 1.2 s | 254 M values/s | 58 MiB |
| HDF5 → CSV, 300 M rows | 2.24 GiB | 5.50 GiB | 27.1 s (+ 5.2 s validation) | 11.1 M values/s | 51 MiB |
| classic → Parquet, 300 M rows | 2.24 GiB | 2.24 GiB | 2.1 s | 146 M values/s | 56 MiB |
| classic → NetCDF-4, deflate level 1 + shuffle, 536.9 M int16 | 1.00 GiB | 0.06 GiB | 29.3 s | 18.3 M values/s | 58 MiB |
| NetCDF-4 (deflate, chunked) → CDF-5, 536.9 M int16 | 0.06 GiB | 1.00 GiB | 20.1 s | 26.6 M values/s | 85 MiB |
| NetCDF-4 (deflate) → Parquet long, 4096 × 16384 subset, 67.1 M rows × 3 columns | 0.06 GiB | 1.25 GiB | 3.4 s | 19.7 M rows/s | 121 MiB |
The synthetic data is smooth or pseudo-random on purpose; real compression ratios and deflate speeds differ.
Untrusted files
- Sizes: every shape, count and length read from a file is checked (overflow-checked products) before memory
is allocated; single reads are bounded (1 GiB, and 2³¹ elements); variable-length data is sized with
H5Dvlen_get_buf_sizebefore being read; attributes over 16 MiB are neither displayed nor copied; ≤ 2 million links per group, ≤ 100 000 attributes per object, ≤ 64 levels, ≤ 200 000 objects per walk; a group reachable through several hard links (how HDF5 files form cycles) is entered once. - Nothing outside the file is followed: external links are listed with their target and never traversed —
the library's external-link class is replaced by one whose traversal always fails, and every access property
list used on a source carries a refusing callback (
HDF5InspectTests.linksAreReportedAndNeverFollowedchecks both, including raw opens with default property lists). Datasets with external raw storage and virtual datasets are flagged and refused (RepackTests.storageThatIsNeverFollowed). - No plug-ins: plug-in loading is disabled at build time (non-existent default directory) and at run time
(
HDF5_PLUGIN_PRELOAD=::, loading state 0, empty search table;HDF5InspectTests.libraryIsHardened). Datasets compressed with a filter the build does not have (szip, blosc, zstd, lzf…) are reported as unreadable by name. - Errors: the library's error stack is captured (never printed) and turned into
ConversionError. - The C library can still hang or crash on a damaged file. In the corruption sweep (600 mutated copies of two
fixtures, deterministic seed) one mutant makes HDF5 2.2.0 loop forever in
H5HG__cache_heap_deserialize. For that reason the package ships a helper executable,orix-hdf5-probe: when it is present in the app'sContents/Helpers(orEmbeddedEngines/tools),inspect,convertandvalidatefirst let it open the file in a process of its own and walk everything the engine reads before converting (tree, dimension scales, every attribute, the first and last block of every dataset), with a 60 s deadline; a crash or a stall there makes the engine refuse the file with a clear message instead of dying. Results are cached per file (path, size, date).HDF5RobustnessTests.corruptedCopiesAreContainedandtruncatedFilesrun 717 damaged files through the helper: 527 + 117 are refused with ordinary errors, 72 are still fully readable, 1 stalls the helper and is contained, none read a truncated file as complete. - NetCDF classic: own parser;
ClassicRobustnessTestscovers malformed headers, every truncation length and several thousand byte-flipped or randomly damaged files across the three variants, in-process (that sweep found and fixed an arithmetic overflow on a hostile record count).
Residual risk
- If
orix-hdf5-probeis not bundled, HDF5 files are read in the application's process and a crafted file can hang or crash it (the engine logs that the helper is absent). The helper is a product of this package (7.8 MB, 5.0 MB stripped, system libraries only) and has to be copied into the app by the lead. - The helper covers the metadata and the first and last block of each dataset. A conversion then reads all the data in-process; damage confined to later chunks is not pre-checked.
preflightis synchronous: for an HDF5 file it opens the file only after the isolated check has passed (or when no helper exists); before that it returns no warning.- A stalled read cannot be interrupted from Swift: cancellation takes effect between blocks.
- All HDF5 work in the process is serialised by one lock (the library is built without thread safety), taken per block: two HDF5 jobs interleave rather than run in parallel.
Evaluated and not implemented
| Item | Why |
|---|---|
| Arrow IPC as a target | not written: it needs its own FlatBuffers writer; Parquet covers typed columnar output. The data engine's parquet → arrow path can follow |
| CSV / Parquet / JSON → HDF5 / NetCDF (tabular sources, grid reshaping) | not implemented in the time available; no such path is declared |
| Parquet and Arrow as sources | the engine's Parquet reader only decodes what its own writer produces (it exists for validation) |
| netcdf-c | see "Why no netCDF library" |
| szip / libaec | not built (kept the dependency set to zlib); szip datasets are reported as unreadable |
| Third-party filters (blosc, zstd, lzf, bitshuffle…) | would mean bundling each codec; plug-in loading stays disabled |
| String variables → NetCDF classic (as character arrays) | refused with an explanation rather than restructured |
| Array-typed datasets → NetCDF (as extra dimensions) | left out with a warning |
| Time decoding for non-standard calendars, month / year units, dates before 1582 in the mixed calendar | values stay numbers, with a warning |
| Applying the HDF5 fill-value property as "missing" | only _FillValue / missing_value attributes are honoured, as CF does |
| Region and object references, HDF5 time class | reported; never converted |
| Process isolation for the whole conversion | only the pre-check runs in the helper (see residual risk) |
| Reading all data in the pre-check | would double the time of large conversions |
| Full re-parse of CSV output for validation | the CSV check counts records and compares the first 2000 rows with a fresh read of the source |
Validation
- Tables: the plan is rebuilt from the source; the row count must match (
sci.row-count); the first rows (up to 2000) are regenerated from the source and compared — byte prefix for CSV, decoded values for Parquet (footer, column names, row-group totals and the recorded transformation are checked too), parsed values for JSON (files up to 256 MiB; nested shape walked). - HDF5 / NetCDF outputs: re-opened; number of dataset names; for each dataset (up to 2000) the shape and two sampled hyperslabs (origin and far corner) against the source; attribute counts.
Known defects and limits
- NetCDF-4 → classic: dimensions are ordered by first use, and dimensions no variable uses are lost (they are placeholders the reader hides); classic → classic keeps both.
- Files written with the NetCDF-4 conventions carry
_NCProperties = version=2,orix_convert=1,hdf5=2.2.0(no netCDF library version, since none wrote the file); netcdf-c 4.10.1 accepts it. - Attributes are listed in name order for HDF5 sources; variables keep creation order only when the file tracks it.
- Space-padded fixed strings are trimmed in tables and previews; copies between HDF5 files keep the stored bytes.
inspectdescribes at most 300 datasets and walks at most 5000 objects (it says so when it stops).- During development one run of the corruption sweep counted a helper crash that the ten later runs did not reproduce; a file-descriptor bug in the classic reader (since fixed) was still present in the test process at that time, and the cause was not established.