SPB Git

Repository Public License

scientific.md 22.3 KB

# Scientific data engine (scientific) — compatibility

Package Packages/EngineScientific, module EngineScientific, factory makeEngines(). HDF5 and NetCDF-4 go through the bundled HDF5 2.2.0 C library (EmbeddedEngines/hdf5, BSD 3-Clause, static, see its NOTICE.md) behind a C shim (Sources/CHDF5Shim). NetCDF classic (CDF-1, CDF-2, CDF-5) is read and written by the engine's own Swift code. CSV, JSON and Parquet output are written by the engine itself (no DuckDB dependency).

State on 2026-11-21 (macOS 27.0, arm64): 10 declared paths, 48 tests, all passing (swift test: Test run with 48 tests in 12 suites passed; one of the 48 is a development aid that only runs when an environment variable is set). The full suite was run 10 times in a row with identical results.

# Why no netCDF library

NetCDF-4 files are HDF5 files that follow conventions; classic files are a small, fully specified binary format. Building netcdf-c would add a second large C parser in front of untrusted files (plus DAP, Zarr and plug-in code to switch off) for no capability the engine lacks. Instead:

  • classic: own reader and writer from the format specification, bounds-checked, tested against files assembled byte by byte in the tests (NetCDFInspectTests.classicFromIndependentBytes, three variants) — and the writer must reproduce those bytes exactly (NetCDFConversionTests.classicWriterReproducesIndependentBytes);
  • NetCDF-4: read and written through HDF5 directly, honouring the NetCDF-4 conventions — dimension scales, dimension-only placeholder datasets (This is a netCDF dimension but not a netCDF variable…), _Netcdf4Dimid, _NCProperties, the _nc4_non_coord_ prefix, creation-order tracking, unlimited dimensions, _FillValue. Tested against a NetCDF-4 file built with raw dimension-scale calls (NetCDFInspectTests.netCDF4BuiltWithRawCalls) and by inspecting the engine's output with the C library (HDF5ToNetCDFTests.everythingToNetCDF4).

During development the engine's outputs were opened with netCDF4-python 1.7.5 (netcdf-c 4.10.1), h5py 3.16, pyarrow 26.0.1 and DuckDB 2.0.1 in a scratch environment: netcdf-c reads the hand-assembled classic fixtures and every NetCDF file the engine writes (dimensions, unlimited flags, attributes, filters, values); h5py reads the HDF5 outputs; pyarrow and DuckDB read the Parquet files with the right types, nulls and key/value metadata. That check found two defects that are now handled (a dataset reachable through a hard link in another group, and compound types with text members, both of which netcdf-c cannot read). None of those tools is used by the tests or shipped.

# Declared paths

Source Destination Operation Status Fidelity Preserves Tests
hdf5 csv — supported partial — FlattenCSVTests (9 tests)
hdf5 parquet — supported partial metadata FlattenParquetTests.typesNullsAndMetadata, severalRowGroups
hdf5 json — supported partial metadata FlattenJSONTests.records, nestedArrays
netcdf csv — supported partial — FlattenCSVTests.netCDFSources (classic and NetCDF-4)
netcdf parquet — supported partial metadata EngineSurfaceTests.netCDFToParquet (classic and NetCDF-4)
netcdf json — supported partial metadata FlattenJSONTests.nestedArrays (classic)
hdf5 netcdf — supported structural metadata, attributes, dimensions HDF5ToNetCDFTests.everythingToNetCDF4, classicVariantsFromHDF5, classicRefusals
netcdf hdf5 — supported structural metadata, attributes, dimensions NetCDFConversionTests.classicToNetCDF4AndBack, netCDF4ToHDF5AndRecompression
netcdf netcdf netcdf.reformat (Change NetCDF Variant) supported structural metadata, attributes, dimensions classicWriterReproducesIndependentBytes, classicToNetCDF4AndBack, netCDF4ToHDF5AndRecompression, severalUnlimitedDimensionsToClassic
hdf5 hdf5 hdf5.repack (Repack / Recompress HDF5) supported structural metadata, attributes, dimensions RepackTests.repackKeepsEverythingItCan, cancellationAndProgress

netcdf covers the three classic variants and NetCDF-4; the container is recognised from the content. Limitations shown to the user: tabular outputs are a flattened view of the selected variables, not the file; CSV has no place for attributes; JSON turns NaN / infinity into null; references, bitfields, half floats and the HDF5 time class have no NetCDF equivalent; classic output is refused for groups, strings and user-defined types; external links, external raw storage and virtual datasets are never followed; szip and third-party filters are not available.

# Inspection

inspect returns the container, the numbers of groups and datasets, the dimensions (name, length, unlimited), global attributes with values, links that are not followed, and one line per dataset: type, shape with dimension names, chunk shape, filters with the deflate level, fill value, coordinate-variable flag, CF hints exactly as read (units, standard_name, calendar, packing) and the attribute count (first 300 datasets, 5000 objects walked).

The richer API for the app's dataset browser is SciFile:

Call Result
SciFile.open(url) (async; runs the isolated check when the helper is bundled) / SciFile(url:) the file, whatever its container
children(of:offset:limit:) lazy tree, paged; creation order where the file tracks it
node(at:) kind (group, dataset, named type, external link with its target, dangling link), SciDatasetInfo, attribute count, soft-link target
SciDatasetInfo type (integers of every width and signedness, floats, fixed / variable strings, char, compound, enum, array, vlen, reference, opaque, bitfield, time), dimensions (name, length, unlimited, coordinate variable, defining scale), layout, chunk shape, filters (with availability), fill value, stored bytes, why it cannot be read if so
attributes(of:offset:limit:maxElements:), attribute(_:of:offset:maxElements:) paged attributes, each with a bounded number of values (attributes over 16 MiB are reported, not loaded)
conventions(of:) units, standard_name, long_name, calendar, axis, scale_factor, add_offset, _FillValue, missing_value
preview(_:start:count:stride:maxElements:) a bounded rectangular slice of any dataset, as text; compound members as columns
tree(maxNodes:) bounded depth-first listing

Dimension-scale bookkeeping (CLASS, NAME on scales, DIMENSION_LIST, REFERENCE_LIST) and NetCDF-4 internals (_NCProperties, _Netcdf4Dimid, _Netcdf4Coordinates, dimension placeholders) are presented as dimensions, not as attributes or datasets. Tests: HDF5InspectTests (8), NetCDFInspectTests (2).

# Flattening to tables (CSV, Parquet, JSON)

Flattening is explicit. Nothing is converted until the engine knows which variables, which index ranges and which layout; the exact row count is computed before any data is read.

  • Variables (variables): dataset paths separated by commas. Left empty, the conversion proceeds only when the file holds exactly one candidate (a dataset of a tabular type that is not a coordinate variable or a soft link); otherwise it stops with an invalidOption error listing the candidates with type and shape — preflight returns the same text as a warning (sci-selection-needed) for the app to present. Several variables must share the same dimensions.
  • Index ranges (subset): dimension=start:stop:stride, comma-separated; stop exclusive, 0-based, parts optional (time=10:, lat=::2, level=3). Unnamed dimensions are dim0, dim1…, or their position.
  • Layout (layout):
Layout Rows Columns
long one per selected cell, C order (last dimension fastest) one per dimension — the coordinate variable's values when there is one, else the index — then one per variable (one per member for a compound)
wide one per index of the first dimension first dimension, then one column per index of the second, named dim=label; exactly one 2-D variable; ≤ 10 000 columns
table one per element one per variable or compound member, no dimension column; 1-D variables and scalars
auto (default) table for a scalar or a single dimension without coordinate values (or when the coordinate variable is itself selected); long otherwise
  • Fill values (fill): keep (default) or null — cells equal to _FillValue / missing_value become null (integers compared as integers, float32 at float32 precision, NaN fills recognised). The HDF5 fill-value property alone is not treated as "missing".
  • Packed values (packed): raw (default) leaves the stored numbers; unpack writes value × scale_factor + add_offset as float64. With raw, a caution warning says the table does not hold the physical quantity.
  • Time (time.decode, off by default): <unit> since <date> numbers become ISO 8601 UTC text, for the standard, gregorian and proleptic_gregorian calendars only; units from microseconds to days. Other calendars (noleap, 360_day, julian…), month / year units and dates before 15 October 1582 in the mixed calendar stay numbers, with a warning naming the calendar and the units.
  • Row limit (rows.limit, default 10 000 000): the conversion is refused above it; raising the option is the confirmation.
  • Text stored as characters along a last dimension (NetCDF char) is folded into one string per row. Enumerations are written as their names, array-typed elements as [a,b,c] text.

Example (/chunked/grid, int16 20 × 30 with scales y, x; options subset = y=0:2, x=0:3):

text
long                      wide (layout=wide, x=0:4, y=0:3)      long, fill=null, packed=unpack
y,x,grid                  y,x=10.0,x=10.5,x=11.0,x=11.5         y,x,grid
0,10.0,-999               0,-999,1,2,3                          0,10.0,
0,10.5,1                  2,100,101,102,103                     0,10.5,273.15999999999997
0,11.0,2                  4,200,201,202,203                     0,11.0,273.16999999999996
2,10.0,100

The transformation is recorded, not only applied: in the facts (sci.transformation, sci.layout, sci.rows, sci.nulls…), in a sci-flattened warning before and during conversion, and in the file:

Target Where
Parquet key/value metadata orix_convert.transformation, orix_convert.attributes, orix_convert.source (JSON text)
JSON the metadata object; records shape {"metadata", "columns", "rows":[{…}]} or nested shape {"metadata", "variable", "dimensions", "shape", "data": nested arrays} (one non-compound variable)
CSV nowhere by default (caution warning counting the attributes); with metadata.sidecar the result is a folder holding name.csv and name.metadata.json

The transformation record lists the source container, layout, row order, row count, each variable (path, type, dimensions, shape, units, calendar, what was done with packing, fill values and time) and each dimension (length, start, count, stride, whether labels are indices or the values of which variable).

Parquet files: one row group per block of at most 1 048 576 rows; PLAIN encoding; exact integer widths and signedness (INT_8…UINT_64 annotations), FLOAT, DOUBLE, UTF-8 strings; nullable columns only where nulls can occur; uncompressed by default, gzip pages on request. No dictionary encoding, statistics or page index.

# Structure-preserving conversions

From → to Carried Refused or reported
HDF5 → NetCDF-4 groups, datasets with their exact types (incl. enum, opaque, variable-length, compound of numbers), attributes, chunk shapes, deflate / shuffle / Fletcher-32, fill values, unlimited dimensions, dimension scales as named dimensions left out with one dataLoss warning each: datasets holding references, bitfields, half or extended floats, the time class, array-typed datasets, compounds with text / enum / vlen / opaque members. Dimensions without a scale → phony_dim_N. A scale outside the variable's group hierarchy → generated dimension + warning. Hard-link aliases and soft links are not NetCDF notions (warning)
HDF5 → NetCDF classic root-level numeric and character datasets, attributes refused: any dataset inside a group, strings, compound / enum / vlen / opaque / array types. Unsigned and 64-bit integers need CDF-5 (chosen automatically, or refused if an older variant is forced). Only one unlimited dimension, leading its variables; others become fixed (warning). String-array attributes joined line by line (warning). Names adapted to NetCDF rules (warning)
NetCDF classic → NetCDF-4 / HDF5 every dimension (also unused ones), variable, attribute, the record dimension as unlimited, _FillValue also as the HDF5 fill value; optional deflate + shuffle nothing
NetCDF-4 → HDF5 same as HDF5 → HDF5; the file keeps its NetCDF-4 dimension machinery and stays readable as NetCDF-4 same as repack
NetCDF-4 → classic as HDF5 → classic as HDF5 → classic; chunking and compression do not exist (info warning)
classic → classic (variant change) byte-identical when the variant is unchanged; otherwise same content content that does not fit the chosen variant is refused, never truncated
HDF5 → HDF5 (repack) everything above plus HDF5-only types (compound with text, arrays, half floats, bitfields), hard links and soft links recreated, named types datasets holding references are left out; external links and dangling soft links are not recreated; filters other than deflate / shuffle / Fletcher-32 are not re-applied (values kept)

compression: preserve (default), none, deflate with compression.level 1–9 and compression.shuffle. netcdf.format: auto (NetCDF-4 from HDF5 or classic; the smallest fitting classic variant from NetCDF-4), netcdf4, classic, 64bit-offset, cdf5, classic-auto.

scale_factor / add_offset and fill values are attributes like any other in these conversions: copied, never applied.

# Large data

Data moves in rectangular blocks of at most 1 048 576 elements (or 16 MiB), aligned to the chunk grid when the source is chunked; progress is the measured share of elements written (reported every 0.5 %); cancellation is checked between blocks. Measured on an M5 Max (48 GB), release build, EngineScientificBench (each line a separate process; peak resident memory from getrusage; validation included in the peak):

Run Source Output Convert Throughput Peak RSS
classic (CDF-5) → HDF5, 300 M float64 2.24 GiB 2.24 GiB 1.1 s 280 M values/s 42 MiB
HDF5 → Parquet, 300 M rows 2.24 GiB 2.24 GiB (287 row groups) 1.2 s 254 M values/s 58 MiB
HDF5 → CSV, 300 M rows 2.24 GiB 5.50 GiB 27.1 s (+ 5.2 s validation) 11.1 M values/s 51 MiB
classic → Parquet, 300 M rows 2.24 GiB 2.24 GiB 2.1 s 146 M values/s 56 MiB
classic → NetCDF-4, deflate level 1 + shuffle, 536.9 M int16 1.00 GiB 0.06 GiB 29.3 s 18.3 M values/s 58 MiB
NetCDF-4 (deflate, chunked) → CDF-5, 536.9 M int16 0.06 GiB 1.00 GiB 20.1 s 26.6 M values/s 85 MiB
NetCDF-4 (deflate) → Parquet long, 4096 × 16384 subset, 67.1 M rows × 3 columns 0.06 GiB 1.25 GiB 3.4 s 19.7 M rows/s 121 MiB

The synthetic data is smooth or pseudo-random on purpose; real compression ratios and deflate speeds differ.

# Untrusted files

  • Sizes: every shape, count and length read from a file is checked (overflow-checked products) before memory is allocated; single reads are bounded (1 GiB, and 2³¹ elements); variable-length data is sized with H5Dvlen_get_buf_size before being read; attributes over 16 MiB are neither displayed nor copied; ≤ 2 million links per group, ≤ 100 000 attributes per object, ≤ 64 levels, ≤ 200 000 objects per walk; a group reachable through several hard links (how HDF5 files form cycles) is entered once.
  • Nothing outside the file is followed: external links are listed with their target and never traversed — the library's external-link class is replaced by one whose traversal always fails, and every access property list used on a source carries a refusing callback (HDF5InspectTests.linksAreReportedAndNeverFollowed checks both, including raw opens with default property lists). Datasets with external raw storage and virtual datasets are flagged and refused (RepackTests.storageThatIsNeverFollowed).
  • No plug-ins: plug-in loading is disabled at build time (non-existent default directory) and at run time (HDF5_PLUGIN_PRELOAD=::, loading state 0, empty search table; HDF5InspectTests.libraryIsHardened). Datasets compressed with a filter the build does not have (szip, blosc, zstd, lzf…) are reported as unreadable by name.
  • Errors: the library's error stack is captured (never printed) and turned into ConversionError.
  • The C library can still hang or crash on a damaged file. In the corruption sweep (600 mutated copies of two fixtures, deterministic seed) one mutant makes HDF5 2.2.0 loop forever in H5HG__cache_heap_deserialize. For that reason the package ships a helper executable, orix-hdf5-probe: when it is present in the app's Contents/Helpers (or EmbeddedEngines/tools), inspect, convert and validate first let it open the file in a process of its own and walk everything the engine reads before converting (tree, dimension scales, every attribute, the first and last block of every dataset), with a 60 s deadline; a crash or a stall there makes the engine refuse the file with a clear message instead of dying. Results are cached per file (path, size, date). HDF5RobustnessTests.corruptedCopiesAreContained and truncatedFiles run 717 damaged files through the helper: 527 + 117 are refused with ordinary errors, 72 are still fully readable, 1 stalls the helper and is contained, none read a truncated file as complete.
  • NetCDF classic: own parser; ClassicRobustnessTests covers malformed headers, every truncation length and several thousand byte-flipped or randomly damaged files across the three variants, in-process (that sweep found and fixed an arithmetic overflow on a hostile record count).

# Residual risk

  • If orix-hdf5-probe is not bundled, HDF5 files are read in the application's process and a crafted file can hang or crash it (the engine logs that the helper is absent). The helper is a product of this package (7.8 MB, 5.0 MB stripped, system libraries only) and has to be copied into the app by the lead.
  • The helper covers the metadata and the first and last block of each dataset. A conversion then reads all the data in-process; damage confined to later chunks is not pre-checked.
  • preflight is synchronous: for an HDF5 file it opens the file only after the isolated check has passed (or when no helper exists); before that it returns no warning.
  • A stalled read cannot be interrupted from Swift: cancellation takes effect between blocks.
  • All HDF5 work in the process is serialised by one lock (the library is built without thread safety), taken per block: two HDF5 jobs interleave rather than run in parallel.

# Evaluated and not implemented

Item Why
Arrow IPC as a target not written: it needs its own FlatBuffers writer; Parquet covers typed columnar output. The data engine's parquet → arrow path can follow
CSV / Parquet / JSON → HDF5 / NetCDF (tabular sources, grid reshaping) not implemented in the time available; no such path is declared
Parquet and Arrow as sources the engine's Parquet reader only decodes what its own writer produces (it exists for validation)
netcdf-c see "Why no netCDF library"
szip / libaec not built (kept the dependency set to zlib); szip datasets are reported as unreadable
Third-party filters (blosc, zstd, lzf, bitshuffle…) would mean bundling each codec; plug-in loading stays disabled
String variables → NetCDF classic (as character arrays) refused with an explanation rather than restructured
Array-typed datasets → NetCDF (as extra dimensions) left out with a warning
Time decoding for non-standard calendars, month / year units, dates before 1582 in the mixed calendar values stay numbers, with a warning
Applying the HDF5 fill-value property as "missing" only _FillValue / missing_value attributes are honoured, as CF does
Region and object references, HDF5 time class reported; never converted
Process isolation for the whole conversion only the pre-check runs in the helper (see residual risk)
Reading all data in the pre-check would double the time of large conversions
Full re-parse of CSV output for validation the CSV check counts records and compares the first 2000 rows with a fresh read of the source

# Validation

  • Tables: the plan is rebuilt from the source; the row count must match (sci.row-count); the first rows (up to 2000) are regenerated from the source and compared — byte prefix for CSV, decoded values for Parquet (footer, column names, row-group totals and the recorded transformation are checked too), parsed values for JSON (files up to 256 MiB; nested shape walked).
  • HDF5 / NetCDF outputs: re-opened; number of dataset names; for each dataset (up to 2000) the shape and two sampled hyperslabs (origin and far corner) against the source; attribute counts.

# Known defects and limits

  • NetCDF-4 → classic: dimensions are ordered by first use, and dimensions no variable uses are lost (they are placeholders the reader hides); classic → classic keeps both.
  • Files written with the NetCDF-4 conventions carry _NCProperties = version=2,orix_convert=1,hdf5=2.2.0 (no netCDF library version, since none wrote the file); netcdf-c 4.10.1 accepts it.
  • Attributes are listed in name order for HDF5 sources; variables keep creation order only when the file tracks it.
  • Space-padded fixed strings are trimmed in tables and previews; copies between HDF5 files keep the stored bytes.
  • inspect describes at most 300 datasets and walks at most 5000 objects (it says so when it stops).
  • During development one run of the corruption sweep counted a helper crash that the ten later runs did not reproduce; a file-descriptor bug in the classic reader (since fixed) was still present in the test process at that time, and the cause was not established.
Files main
Menu

Theme