SPB Git

Repository Public License

data.md 69.8 KB

# Data engine (data) — spreadsheets, tables and structured documents

Package Packages/EngineData (module EngineData, factory makeEngines()), engine id data, display name "Spreadsheets & Data". Everything below was implemented, run and measured on 2026-11-21 on this Mac (Apple M-series, 18 cores, 48 GB, macOS 27.0, Swift 6.4.2). Nothing is listed that is not covered by a passing test.

Formats: XLSX, XLS, ODS, CSV, TSV, JSON, JSON Lines, Parquet, Arrow IPC (file = Feather v2, and stream), SQLite, DuckDB database files, XML, YAML, TOML, property lists (XML and binary).

# 1. Architecture in one page

  • DuckDB v2.0.1 (official libduckdb, arm64, C API) is the tabular hub. Every tabular conversion is "source → DuckDB relation → target":
    • CSV/TSV, JSON/JSON Lines, Parquet and DuckDB files are scanned by DuckDB itself (streaming, multi-threaded, bounded by memory_limit, spilling to a temp_directory inside the job's work directory).
    • XLSX, ODS, XLS, SQLite and Arrow are read by the engine's own readers and staged, row by row, into a DuckDB table that lives in a database file inside the work directory (so they are not limited by memory either). Each column records the kinds of values actually seen and is cast to the narrowest type that holds them all exactly.
    • XML, YAML, TOML, property lists and JSON objects are parsed into an ordered value tree; the list of records is located and handed to DuckDB.
    • CSV/TSV/JSON/JSON Lines/Parquet are written by DuckDB COPY, DuckDB files by CREATE TABLE AS; everything else is written by the engine's own writers from streamed result chunks.
  • Tree conversions (JSON, YAML, TOML, XML, plist ↔ each other) never go through a table: parse → ordered tree → serialise.
  • Sandbox. Each job opens its own DuckDB session with autoinstall_known_extensions=false, autoload_known_extensions=false, enable_external_access=false, allowed_directories = the job's private workspace + the work directory, allowed_paths = the job's source file, TimeZone='UTC', then lock_configuration=true. SQL can therefore not read or write any other file, cannot install or load an extension and cannot undo these settings (test duckdbIsLockedDown: 23 escape attempts, all refused). No user text is ever spliced into SQL unquoted: names and paths go through SQL.ident / SQL.literal (test hostileFileSheetAndColumnNames). A source file whose name contains glob characters (* ? [ ] { }) is reached through a hard link with a neutral name, so the name can never match another file.
  • What the bundled DuckDB build contains (verified by running it): core functions, CSV, parquet, json, icu. Its SQLite, Excel, Arrow and spatial support live in downloadable extensions and are not used.
  • The pipeline is synchronous and runs on a dedicated 8 MB-stack thread per job; task cancellation interrupts the running DuckDB statement / SQLite statement / LibreOffice process and is checked in every row loop.
  • Progress is measured: DuckDB's own scan progress for COPY/verification passes, compressed bytes consumed for XLSX/ODS, rows written over rows staged for streamed targets, file offset for Arrow. A job declares how many full passes over the data it makes (1 or 2) and reports (finished passes + fraction of the current one) / passes. Steps that cannot be measured (LibreOffice, tree documents) report fraction: nil.

# Bundled / used components

Component Version License How Size
DuckDB libduckdb.dylib v2.0.1 (pinned, SHA-256 in EmbeddedEngines/duckdb/build.sh) MIT dynamic library in CDuckDB.xcframework, SwiftPM binaryTarget 62 MB (arm64)
Yams 6.2.2 (exact pin) MIT SwiftPM source dependency (wraps libyaml, compiled by SwiftPM) source
SQLite system libsqlite3 (3.54.0 on this Mac) public domain linked from the macOS SDK —
Compression, zlib (crc32 only) system — Apple frameworks / SDK —
LibreOffice (soffice) bundled by the documents engine in EmbeddedEngines/tools/LibreOffice.app MPL-2.0 separate process through ProcessRunner, XLS only not owned by this engine

Written for this engine (no third-party code): ZIP reader/writer, streaming XML tokenizer, XLSX reader/writer, ODS reader/writer, Arrow IPC reader/writer (FlatBuffers metadata + LZ4-frame decoder), exact JSON parser/serialiser, TOML 1.0 parser/serialiser, XML ↔ tree mapping, text-encoding detection/transcoding, delimiter sniffer, column type detection.

# 2. Declared paths

219 paths are declared when the bundled LibreOffice is present (as today), 191 otherwise: the 28 paths that read or write XLS exist only with it (paths is computed; convert answers engineUnavailable if XLS is requested without it). No named operations; every path is a plain format conversion. Same-format paths exist only where re-writing with other options is meaningful and safe: CSV, TSV, JSON Lines, Parquet, JSON, YAML, TOML, XML, plist.

Status is supported for all of them: each one converts a fixture, passes the engine's own validate, and is read back and compared value by value in the parameterised tests MatrixTests.path (194 tabular paths) and TreeTests.path (25 tree paths).

Fidelity as declared

  • lossless: Parquet → Parquet, Parquet → DuckDB, JSON → JSON, JSON → YAML, plist → plist. The output reads back identical (types and values; for plist, the same property list).
  • partial: every path from a workbook (XLSX/XLS/ODS: only cell values are carried), and every path from a database file to a single-table target (one table per output file).
  • structural: everything else — the content is kept but re-expressed (types detected from text, dates as ISO text, nested values flattened or stringified…). On these paths the canonical dataset of the tests still reads back with equal values (compared as numbers / instants / text); the differences are type re-expression, documented below.
Full path table (219 rows)
Source Destination Operation Status Fidelity Preserves Backend (read → write) Test
xlsx xls — supported partial multipleSheets, tables engine XLSX reader → staging → engine XLSX writer → LibreOffice MatrixTests.path (xlsx → xls)
xlsx ods — supported partial multipleSheets, tables engine XLSX reader → staging → engine ODS writer MatrixTests.path (xlsx → ods)
xlsx csv — supported partial tables engine XLSX reader → staging → DuckDB COPY … (FORMAT csv) MatrixTests.path (xlsx → csv)
xlsx tsv — supported partial tables engine XLSX reader → staging → DuckDB COPY … (FORMAT csv) MatrixTests.path (xlsx → tsv)
xlsx json — supported partial tables engine XLSX reader → staging → DuckDB COPY … (FORMAT json) MatrixTests.path (xlsx → json)
xlsx jsonl — supported partial tables engine XLSX reader → staging → DuckDB COPY … (FORMAT json) MatrixTests.path (xlsx → jsonl)
xlsx parquet — supported partial tables engine XLSX reader → staging → DuckDB COPY … (FORMAT parquet) MatrixTests.path (xlsx → parquet)
xlsx arrow — supported partial tables engine XLSX reader → staging → engine Arrow IPC writer MatrixTests.path (xlsx → arrow)
xlsx sqlite — supported partial multipleSheets, tables engine XLSX reader → staging → system libsqlite3 MatrixTests.path (xlsx → sqlite)
xlsx duckdb — supported partial multipleSheets, tables engine XLSX reader → staging → DuckDB CREATE TABLE AS MatrixTests.path (xlsx → duckdb)
xlsx xml — supported partial tables engine XLSX reader → staging → engine XML writer MatrixTests.path (xlsx → xml)
xlsx yaml — supported partial tables engine XLSX reader → staging → Yams (row by row) MatrixTests.path (xlsx → yaml)
xlsx toml — supported partial tables engine XLSX reader → staging → engine TOML writer MatrixTests.path (xlsx → toml)
xlsx plist — supported partial tables engine XLSX reader → staging → PropertyListSerialization MatrixTests.path (xlsx → plist)
xls xlsx — supported partial multipleSheets, tables LibreOffice → XLSX → engine XLSX reader → engine XLSX writer MatrixTests.path (xls → xlsx)
xls ods — supported partial multipleSheets, tables LibreOffice → XLSX → engine XLSX reader → engine ODS writer MatrixTests.path (xls → ods)
xls csv — supported partial tables LibreOffice → XLSX → engine XLSX reader → DuckDB COPY … (FORMAT csv) MatrixTests.path (xls → csv)
xls tsv — supported partial tables LibreOffice → XLSX → engine XLSX reader → DuckDB COPY … (FORMAT csv) MatrixTests.path (xls → tsv)
xls json — supported partial tables LibreOffice → XLSX → engine XLSX reader → DuckDB COPY … (FORMAT json) MatrixTests.path (xls → json)
xls jsonl — supported partial tables LibreOffice → XLSX → engine XLSX reader → DuckDB COPY … (FORMAT json) MatrixTests.path (xls → jsonl)
xls parquet — supported partial tables LibreOffice → XLSX → engine XLSX reader → DuckDB COPY … (FORMAT parquet) MatrixTests.path (xls → parquet)
xls arrow — supported partial tables LibreOffice → XLSX → engine XLSX reader → engine Arrow IPC writer MatrixTests.path (xls → arrow)
xls sqlite — supported partial multipleSheets, tables LibreOffice → XLSX → engine XLSX reader → system libsqlite3 MatrixTests.path (xls → sqlite)
xls duckdb — supported partial multipleSheets, tables LibreOffice → XLSX → engine XLSX reader → DuckDB CREATE TABLE AS MatrixTests.path (xls → duckdb)
xls xml — supported partial tables LibreOffice → XLSX → engine XLSX reader → engine XML writer MatrixTests.path (xls → xml)
xls yaml — supported partial tables LibreOffice → XLSX → engine XLSX reader → Yams (row by row) MatrixTests.path (xls → yaml)
xls toml — supported partial tables LibreOffice → XLSX → engine XLSX reader → engine TOML writer MatrixTests.path (xls → toml)
xls plist — supported partial tables LibreOffice → XLSX → engine XLSX reader → PropertyListSerialization MatrixTests.path (xls → plist)
ods xlsx — supported partial multipleSheets, tables engine ODS reader → staging → engine XLSX writer MatrixTests.path (ods → xlsx)
ods xls — supported partial multipleSheets, tables engine ODS reader → staging → engine XLSX writer → LibreOffice MatrixTests.path (ods → xls)
ods csv — supported partial tables engine ODS reader → staging → DuckDB COPY … (FORMAT csv) MatrixTests.path (ods → csv)
ods tsv — supported partial tables engine ODS reader → staging → DuckDB COPY … (FORMAT csv) MatrixTests.path (ods → tsv)
ods json — supported partial tables engine ODS reader → staging → DuckDB COPY … (FORMAT json) MatrixTests.path (ods → json)
ods jsonl — supported partial tables engine ODS reader → staging → DuckDB COPY … (FORMAT json) MatrixTests.path (ods → jsonl)
ods parquet — supported partial tables engine ODS reader → staging → DuckDB COPY … (FORMAT parquet) MatrixTests.path (ods → parquet)
ods arrow — supported partial tables engine ODS reader → staging → engine Arrow IPC writer MatrixTests.path (ods → arrow)
ods sqlite — supported partial multipleSheets, tables engine ODS reader → staging → system libsqlite3 MatrixTests.path (ods → sqlite)
ods duckdb — supported partial multipleSheets, tables engine ODS reader → staging → DuckDB CREATE TABLE AS MatrixTests.path (ods → duckdb)
ods xml — supported partial tables engine ODS reader → staging → engine XML writer MatrixTests.path (ods → xml)
ods yaml — supported partial tables engine ODS reader → staging → Yams (row by row) MatrixTests.path (ods → yaml)
ods toml — supported partial tables engine ODS reader → staging → engine TOML writer MatrixTests.path (ods → toml)
ods plist — supported partial tables engine ODS reader → staging → PropertyListSerialization MatrixTests.path (ods → plist)
csv xlsx — supported structural tables DuckDB read_csv + engine type detection → engine XLSX writer MatrixTests.path (csv → xlsx)
csv xls — supported structural tables DuckDB read_csv + engine type detection → engine XLSX writer → LibreOffice MatrixTests.path (csv → xls)
csv ods — supported structural tables DuckDB read_csv + engine type detection → engine ODS writer MatrixTests.path (csv → ods)
csv csv — supported structural tables DuckDB read_csv + engine type detection → DuckDB COPY … (FORMAT csv) MatrixTests.path (csv → csv)
csv tsv — supported structural tables DuckDB read_csv + engine type detection → DuckDB COPY … (FORMAT csv) MatrixTests.path (csv → tsv)
csv json — supported structural tables DuckDB read_csv + engine type detection → DuckDB COPY … (FORMAT json) MatrixTests.path (csv → json)
csv jsonl — supported structural tables DuckDB read_csv + engine type detection → DuckDB COPY … (FORMAT json) MatrixTests.path (csv → jsonl)
csv parquet — supported structural tables DuckDB read_csv + engine type detection → DuckDB COPY … (FORMAT parquet) MatrixTests.path (csv → parquet)
csv arrow — supported structural tables DuckDB read_csv + engine type detection → engine Arrow IPC writer MatrixTests.path (csv → arrow)
csv sqlite — supported structural tables DuckDB read_csv + engine type detection → system libsqlite3 MatrixTests.path (csv → sqlite)
csv duckdb — supported structural tables DuckDB read_csv + engine type detection → DuckDB CREATE TABLE AS MatrixTests.path (csv → duckdb)
csv xml — supported structural tables DuckDB read_csv + engine type detection → engine XML writer MatrixTests.path (csv → xml)
csv yaml — supported structural tables DuckDB read_csv + engine type detection → Yams (row by row) MatrixTests.path (csv → yaml)
csv toml — supported structural tables DuckDB read_csv + engine type detection → engine TOML writer MatrixTests.path (csv → toml)
csv plist — supported structural tables DuckDB read_csv + engine type detection → PropertyListSerialization MatrixTests.path (csv → plist)
tsv xlsx — supported structural tables DuckDB read_csv + engine type detection → engine XLSX writer MatrixTests.path (tsv → xlsx)
tsv xls — supported structural tables DuckDB read_csv + engine type detection → engine XLSX writer → LibreOffice MatrixTests.path (tsv → xls)
tsv ods — supported structural tables DuckDB read_csv + engine type detection → engine ODS writer MatrixTests.path (tsv → ods)
tsv csv — supported structural tables DuckDB read_csv + engine type detection → DuckDB COPY … (FORMAT csv) MatrixTests.path (tsv → csv)
tsv tsv — supported structural tables DuckDB read_csv + engine type detection → DuckDB COPY … (FORMAT csv) MatrixTests.path (tsv → tsv)
tsv json — supported structural tables DuckDB read_csv + engine type detection → DuckDB COPY … (FORMAT json) MatrixTests.path (tsv → json)
tsv jsonl — supported structural tables DuckDB read_csv + engine type detection → DuckDB COPY … (FORMAT json) MatrixTests.path (tsv → jsonl)
tsv parquet — supported structural tables DuckDB read_csv + engine type detection → DuckDB COPY … (FORMAT parquet) MatrixTests.path (tsv → parquet)
tsv arrow — supported structural tables DuckDB read_csv + engine type detection → engine Arrow IPC writer MatrixTests.path (tsv → arrow)
tsv sqlite — supported structural tables DuckDB read_csv + engine type detection → system libsqlite3 MatrixTests.path (tsv → sqlite)
tsv duckdb — supported structural tables DuckDB read_csv + engine type detection → DuckDB CREATE TABLE AS MatrixTests.path (tsv → duckdb)
tsv xml — supported structural tables DuckDB read_csv + engine type detection → engine XML writer MatrixTests.path (tsv → xml)
tsv yaml — supported structural tables DuckDB read_csv + engine type detection → Yams (row by row) MatrixTests.path (tsv → yaml)
tsv toml — supported structural tables DuckDB read_csv + engine type detection → engine TOML writer MatrixTests.path (tsv → toml)
tsv plist — supported structural tables DuckDB read_csv + engine type detection → PropertyListSerialization MatrixTests.path (tsv → plist)
json xlsx — supported structural tables DuckDB read_json (top-level array) / engine JSON parser (object) → engine XLSX writer MatrixTests.path (json → xlsx)
json xls — supported structural tables DuckDB read_json (top-level array) / engine JSON parser (object) → engine XLSX writer → LibreOffice MatrixTests.path (json → xls)
json ods — supported structural tables DuckDB read_json (top-level array) / engine JSON parser (object) → engine ODS writer MatrixTests.path (json → ods)
json csv — supported structural tables DuckDB read_json (top-level array) / engine JSON parser (object) → DuckDB COPY … (FORMAT csv) MatrixTests.path (json → csv)
json tsv — supported structural tables DuckDB read_json (top-level array) / engine JSON parser (object) → DuckDB COPY … (FORMAT csv) MatrixTests.path (json → tsv)
json json — supported lossless engine JSON codec → engine JSON codec TreeTests.path (json → json)
json jsonl — supported structural tables DuckDB read_json (top-level array) / engine JSON parser (object) → DuckDB COPY … (FORMAT json) MatrixTests.path (json → jsonl)
json parquet — supported structural tables DuckDB read_json (top-level array) / engine JSON parser (object) → DuckDB COPY … (FORMAT parquet) MatrixTests.path (json → parquet)
json arrow — supported structural tables DuckDB read_json (top-level array) / engine JSON parser (object) → engine Arrow IPC writer MatrixTests.path (json → arrow)
json sqlite — supported structural tables DuckDB read_json (top-level array) / engine JSON parser (object) → system libsqlite3 MatrixTests.path (json → sqlite)
json duckdb — supported structural tables DuckDB read_json (top-level array) / engine JSON parser (object) → DuckDB CREATE TABLE AS MatrixTests.path (json → duckdb)
json xml — supported structural engine JSON codec → XMLParser / engine XML writer TreeTests.path (json → xml)
json yaml — supported lossless engine JSON codec → Yams TreeTests.path (json → yaml)
json toml — supported structural engine JSON codec → engine TOML codec TreeTests.path (json → toml)
json plist — supported structural engine JSON codec → PropertyListSerialization TreeTests.path (json → plist)
jsonl xlsx — supported structural tables DuckDB read_json → engine XLSX writer MatrixTests.path (jsonl → xlsx)
jsonl xls — supported structural tables DuckDB read_json → engine XLSX writer → LibreOffice MatrixTests.path (jsonl → xls)
jsonl ods — supported structural tables DuckDB read_json → engine ODS writer MatrixTests.path (jsonl → ods)
jsonl csv — supported structural tables DuckDB read_json → DuckDB COPY … (FORMAT csv) MatrixTests.path (jsonl → csv)
jsonl tsv — supported structural tables DuckDB read_json → DuckDB COPY … (FORMAT csv) MatrixTests.path (jsonl → tsv)
jsonl json — supported structural tables DuckDB read_json → DuckDB COPY … (FORMAT json) MatrixTests.path (jsonl → json)
jsonl jsonl — supported structural tables DuckDB read_json → DuckDB COPY … (FORMAT json) MatrixTests.path (jsonl → jsonl)
jsonl parquet — supported structural tables DuckDB read_json → DuckDB COPY … (FORMAT parquet) MatrixTests.path (jsonl → parquet)
jsonl arrow — supported structural tables DuckDB read_json → engine Arrow IPC writer MatrixTests.path (jsonl → arrow)
jsonl sqlite — supported structural tables DuckDB read_json → system libsqlite3 MatrixTests.path (jsonl → sqlite)
jsonl duckdb — supported structural tables DuckDB read_json → DuckDB CREATE TABLE AS MatrixTests.path (jsonl → duckdb)
jsonl xml — supported structural tables DuckDB read_json → engine XML writer MatrixTests.path (jsonl → xml)
jsonl yaml — supported structural tables DuckDB read_json → Yams (row by row) MatrixTests.path (jsonl → yaml)
jsonl toml — supported structural tables DuckDB read_json → engine TOML writer MatrixTests.path (jsonl → toml)
jsonl plist — supported structural tables DuckDB read_json → PropertyListSerialization MatrixTests.path (jsonl → plist)
parquet xlsx — supported structural tables DuckDB read_parquet → engine XLSX writer MatrixTests.path (parquet → xlsx)
parquet xls — supported structural tables DuckDB read_parquet → engine XLSX writer → LibreOffice MatrixTests.path (parquet → xls)
parquet ods — supported structural tables DuckDB read_parquet → engine ODS writer MatrixTests.path (parquet → ods)
parquet csv — supported structural tables DuckDB read_parquet → DuckDB COPY … (FORMAT csv) MatrixTests.path (parquet → csv)
parquet tsv — supported structural tables DuckDB read_parquet → DuckDB COPY … (FORMAT csv) MatrixTests.path (parquet → tsv)
parquet json — supported structural tables DuckDB read_parquet → DuckDB COPY … (FORMAT json) MatrixTests.path (parquet → json)
parquet jsonl — supported structural tables DuckDB read_parquet → DuckDB COPY … (FORMAT json) MatrixTests.path (parquet → jsonl)
parquet parquet — supported lossless tables DuckDB read_parquet → DuckDB COPY … (FORMAT parquet) MatrixTests.path (parquet → parquet)
parquet arrow — supported structural tables DuckDB read_parquet → engine Arrow IPC writer MatrixTests.path (parquet → arrow)
parquet sqlite — supported structural tables DuckDB read_parquet → system libsqlite3 MatrixTests.path (parquet → sqlite)
parquet duckdb — supported lossless tables DuckDB read_parquet → DuckDB CREATE TABLE AS MatrixTests.path (parquet → duckdb)
parquet xml — supported structural tables DuckDB read_parquet → engine XML writer MatrixTests.path (parquet → xml)
parquet yaml — supported structural tables DuckDB read_parquet → Yams (row by row) MatrixTests.path (parquet → yaml)
parquet toml — supported structural tables DuckDB read_parquet → engine TOML writer MatrixTests.path (parquet → toml)
parquet plist — supported structural tables DuckDB read_parquet → PropertyListSerialization MatrixTests.path (parquet → plist)
arrow xlsx — supported structural tables engine Arrow IPC reader → staging → engine XLSX writer MatrixTests.path (arrow → xlsx)
arrow xls — supported structural tables engine Arrow IPC reader → staging → engine XLSX writer → LibreOffice MatrixTests.path (arrow → xls)
arrow ods — supported structural tables engine Arrow IPC reader → staging → engine ODS writer MatrixTests.path (arrow → ods)
arrow csv — supported structural tables engine Arrow IPC reader → staging → DuckDB COPY … (FORMAT csv) MatrixTests.path (arrow → csv)
arrow tsv — supported structural tables engine Arrow IPC reader → staging → DuckDB COPY … (FORMAT csv) MatrixTests.path (arrow → tsv)
arrow json — supported structural tables engine Arrow IPC reader → staging → DuckDB COPY … (FORMAT json) MatrixTests.path (arrow → json)
arrow jsonl — supported structural tables engine Arrow IPC reader → staging → DuckDB COPY … (FORMAT json) MatrixTests.path (arrow → jsonl)
arrow parquet — supported structural tables engine Arrow IPC reader → staging → DuckDB COPY … (FORMAT parquet) MatrixTests.path (arrow → parquet)
arrow sqlite — supported structural tables engine Arrow IPC reader → staging → system libsqlite3 MatrixTests.path (arrow → sqlite)
arrow duckdb — supported structural tables engine Arrow IPC reader → staging → DuckDB CREATE TABLE AS MatrixTests.path (arrow → duckdb)
arrow xml — supported structural tables engine Arrow IPC reader → staging → engine XML writer MatrixTests.path (arrow → xml)
arrow yaml — supported structural tables engine Arrow IPC reader → staging → Yams (row by row) MatrixTests.path (arrow → yaml)
arrow toml — supported structural tables engine Arrow IPC reader → staging → engine TOML writer MatrixTests.path (arrow → toml)
arrow plist — supported structural tables engine Arrow IPC reader → staging → PropertyListSerialization MatrixTests.path (arrow → plist)
sqlite xlsx — supported structural multipleSheets, tables system libsqlite3 → staging → engine XLSX writer MatrixTests.path (sqlite → xlsx)
sqlite xls — supported structural multipleSheets, tables system libsqlite3 → staging → engine XLSX writer → LibreOffice MatrixTests.path (sqlite → xls)
sqlite ods — supported structural multipleSheets, tables system libsqlite3 → staging → engine ODS writer MatrixTests.path (sqlite → ods)
sqlite csv — supported partial tables system libsqlite3 → staging → DuckDB COPY … (FORMAT csv) MatrixTests.path (sqlite → csv)
sqlite tsv — supported partial tables system libsqlite3 → staging → DuckDB COPY … (FORMAT csv) MatrixTests.path (sqlite → tsv)
sqlite json — supported partial tables system libsqlite3 → staging → DuckDB COPY … (FORMAT json) MatrixTests.path (sqlite → json)
sqlite jsonl — supported partial tables system libsqlite3 → staging → DuckDB COPY … (FORMAT json) MatrixTests.path (sqlite → jsonl)
sqlite parquet — supported partial tables system libsqlite3 → staging → DuckDB COPY … (FORMAT parquet) MatrixTests.path (sqlite → parquet)
sqlite arrow — supported partial tables system libsqlite3 → staging → engine Arrow IPC writer MatrixTests.path (sqlite → arrow)
sqlite duckdb — supported structural multipleSheets, tables system libsqlite3 → staging → DuckDB CREATE TABLE AS MatrixTests.path (sqlite → duckdb)
sqlite xml — supported partial tables system libsqlite3 → staging → engine XML writer MatrixTests.path (sqlite → xml)
sqlite yaml — supported partial tables system libsqlite3 → staging → Yams (row by row) MatrixTests.path (sqlite → yaml)
sqlite toml — supported partial tables system libsqlite3 → staging → engine TOML writer MatrixTests.path (sqlite → toml)
sqlite plist — supported partial tables system libsqlite3 → staging → PropertyListSerialization MatrixTests.path (sqlite → plist)
duckdb xlsx — supported structural multipleSheets, tables DuckDB ATTACH (read-only) → engine XLSX writer MatrixTests.path (duckdb → xlsx)
duckdb xls — supported structural multipleSheets, tables DuckDB ATTACH (read-only) → engine XLSX writer → LibreOffice MatrixTests.path (duckdb → xls)
duckdb ods — supported structural multipleSheets, tables DuckDB ATTACH (read-only) → engine ODS writer MatrixTests.path (duckdb → ods)
duckdb csv — supported partial tables DuckDB ATTACH (read-only) → DuckDB COPY … (FORMAT csv) MatrixTests.path (duckdb → csv)
duckdb tsv — supported partial tables DuckDB ATTACH (read-only) → DuckDB COPY … (FORMAT csv) MatrixTests.path (duckdb → tsv)
duckdb json — supported partial tables DuckDB ATTACH (read-only) → DuckDB COPY … (FORMAT json) MatrixTests.path (duckdb → json)
duckdb jsonl — supported partial tables DuckDB ATTACH (read-only) → DuckDB COPY … (FORMAT json) MatrixTests.path (duckdb → jsonl)
duckdb parquet — supported partial tables DuckDB ATTACH (read-only) → DuckDB COPY … (FORMAT parquet) MatrixTests.path (duckdb → parquet)
duckdb arrow — supported partial tables DuckDB ATTACH (read-only) → engine Arrow IPC writer MatrixTests.path (duckdb → arrow)
duckdb sqlite — supported structural multipleSheets, tables DuckDB ATTACH (read-only) → system libsqlite3 MatrixTests.path (duckdb → sqlite)
duckdb xml — supported partial tables DuckDB ATTACH (read-only) → engine XML writer MatrixTests.path (duckdb → xml)
duckdb yaml — supported partial tables DuckDB ATTACH (read-only) → Yams (row by row) MatrixTests.path (duckdb → yaml)
duckdb toml — supported partial tables DuckDB ATTACH (read-only) → engine TOML writer MatrixTests.path (duckdb → toml)
duckdb plist — supported partial tables DuckDB ATTACH (read-only) → PropertyListSerialization MatrixTests.path (duckdb → plist)
xml xlsx — supported structural tables XMLParser → records → engine XLSX writer MatrixTests.path (xml → xlsx)
xml xls — supported structural tables XMLParser → records → engine XLSX writer → LibreOffice MatrixTests.path (xml → xls)
xml ods — supported structural tables XMLParser → records → engine ODS writer MatrixTests.path (xml → ods)
xml csv — supported structural tables XMLParser → records → DuckDB COPY … (FORMAT csv) MatrixTests.path (xml → csv)
xml tsv — supported structural tables XMLParser → records → DuckDB COPY … (FORMAT csv) MatrixTests.path (xml → tsv)
xml json — supported structural XMLParser / engine XML writer → engine JSON codec TreeTests.path (xml → json)
xml jsonl — supported structural tables XMLParser → records → DuckDB COPY … (FORMAT json) MatrixTests.path (xml → jsonl)
xml parquet — supported structural tables XMLParser → records → DuckDB COPY … (FORMAT parquet) MatrixTests.path (xml → parquet)
xml arrow — supported structural tables XMLParser → records → engine Arrow IPC writer MatrixTests.path (xml → arrow)
xml sqlite — supported structural tables XMLParser → records → system libsqlite3 MatrixTests.path (xml → sqlite)
xml duckdb — supported structural tables XMLParser → records → DuckDB CREATE TABLE AS MatrixTests.path (xml → duckdb)
xml xml — supported structural XMLParser / engine XML writer → XMLParser / engine XML writer TreeTests.path (xml → xml)
xml yaml — supported structural XMLParser / engine XML writer → Yams TreeTests.path (xml → yaml)
xml toml — supported structural XMLParser / engine XML writer → engine TOML codec TreeTests.path (xml → toml)
xml plist — supported structural XMLParser / engine XML writer → PropertyListSerialization TreeTests.path (xml → plist)
yaml xlsx — supported structural tables Yams → records → engine XLSX writer MatrixTests.path (yaml → xlsx)
yaml xls — supported structural tables Yams → records → engine XLSX writer → LibreOffice MatrixTests.path (yaml → xls)
yaml ods — supported structural tables Yams → records → engine ODS writer MatrixTests.path (yaml → ods)
yaml csv — supported structural tables Yams → records → DuckDB COPY … (FORMAT csv) MatrixTests.path (yaml → csv)
yaml tsv — supported structural tables Yams → records → DuckDB COPY … (FORMAT csv) MatrixTests.path (yaml → tsv)
yaml json — supported structural Yams → engine JSON codec TreeTests.path (yaml → json)
yaml jsonl — supported structural tables Yams → records → DuckDB COPY … (FORMAT json) MatrixTests.path (yaml → jsonl)
yaml parquet — supported structural tables Yams → records → DuckDB COPY … (FORMAT parquet) MatrixTests.path (yaml → parquet)
yaml arrow — supported structural tables Yams → records → engine Arrow IPC writer MatrixTests.path (yaml → arrow)
yaml sqlite — supported structural tables Yams → records → system libsqlite3 MatrixTests.path (yaml → sqlite)
yaml duckdb — supported structural tables Yams → records → DuckDB CREATE TABLE AS MatrixTests.path (yaml → duckdb)
yaml xml — supported structural Yams → XMLParser / engine XML writer TreeTests.path (yaml → xml)
yaml yaml — supported structural Yams → Yams TreeTests.path (yaml → yaml)
yaml toml — supported structural Yams → engine TOML codec TreeTests.path (yaml → toml)
yaml plist — supported structural Yams → PropertyListSerialization TreeTests.path (yaml → plist)
toml xlsx — supported structural tables engine TOML parser → records → engine XLSX writer MatrixTests.path (toml → xlsx)
toml xls — supported structural tables engine TOML parser → records → engine XLSX writer → LibreOffice MatrixTests.path (toml → xls)
toml ods — supported structural tables engine TOML parser → records → engine ODS writer MatrixTests.path (toml → ods)
toml csv — supported structural tables engine TOML parser → records → DuckDB COPY … (FORMAT csv) MatrixTests.path (toml → csv)
toml tsv — supported structural tables engine TOML parser → records → DuckDB COPY … (FORMAT csv) MatrixTests.path (toml → tsv)
toml json — supported structural engine TOML codec → engine JSON codec TreeTests.path (toml → json)
toml jsonl — supported structural tables engine TOML parser → records → DuckDB COPY … (FORMAT json) MatrixTests.path (toml → jsonl)
toml parquet — supported structural tables engine TOML parser → records → DuckDB COPY … (FORMAT parquet) MatrixTests.path (toml → parquet)
toml arrow — supported structural tables engine TOML parser → records → engine Arrow IPC writer MatrixTests.path (toml → arrow)
toml sqlite — supported structural tables engine TOML parser → records → system libsqlite3 MatrixTests.path (toml → sqlite)
toml duckdb — supported structural tables engine TOML parser → records → DuckDB CREATE TABLE AS MatrixTests.path (toml → duckdb)
toml xml — supported structural engine TOML codec → XMLParser / engine XML writer TreeTests.path (toml → xml)
toml yaml — supported structural engine TOML codec → Yams TreeTests.path (toml → yaml)
toml toml — supported structural engine TOML codec → engine TOML codec TreeTests.path (toml → toml)
toml plist — supported structural engine TOML codec → PropertyListSerialization TreeTests.path (toml → plist)
plist xlsx — supported structural tables PropertyListSerialization → records → engine XLSX writer MatrixTests.path (plist → xlsx)
plist xls — supported structural tables PropertyListSerialization → records → engine XLSX writer → LibreOffice MatrixTests.path (plist → xls)
plist ods — supported structural tables PropertyListSerialization → records → engine ODS writer MatrixTests.path (plist → ods)
plist csv — supported structural tables PropertyListSerialization → records → DuckDB COPY … (FORMAT csv) MatrixTests.path (plist → csv)
plist tsv — supported structural tables PropertyListSerialization → records → DuckDB COPY … (FORMAT csv) MatrixTests.path (plist → tsv)
plist json — supported structural PropertyListSerialization → engine JSON codec TreeTests.path (plist → json)
plist jsonl — supported structural tables PropertyListSerialization → records → DuckDB COPY … (FORMAT json) MatrixTests.path (plist → jsonl)
plist parquet — supported structural tables PropertyListSerialization → records → DuckDB COPY … (FORMAT parquet) MatrixTests.path (plist → parquet)
plist arrow — supported structural tables PropertyListSerialization → records → engine Arrow IPC writer MatrixTests.path (plist → arrow)
plist sqlite — supported structural tables PropertyListSerialization → records → system libsqlite3 MatrixTests.path (plist → sqlite)
plist duckdb — supported structural tables PropertyListSerialization → records → DuckDB CREATE TABLE AS MatrixTests.path (plist → duckdb)
plist xml — supported structural PropertyListSerialization → XMLParser / engine XML writer TreeTests.path (plist → xml)
plist yaml — supported structural PropertyListSerialization → Yams TreeTests.path (plist → yaml)
plist toml — supported structural PropertyListSerialization → engine TOML codec TreeTests.path (plist → toml)
plist plist — supported lossless PropertyListSerialization → PropertyListSerialization TreeTests.path (plist → plist)

# Limitations shown to the user (verbatim from the engine, with the paths they apply to)

  • Only cell values are converted. Formulas (their last calculated values are kept), styles, number formats, column widths, charts, images, pivot tables, comments, data validation and conditional formatting are not carried over. — from ods, xls, xlsx (42 paths)
  • Merged cells keep their value in the first cell only. — from ods, xls, xlsx (42 paths)
  • Spreadsheet numbers are 64-bit floats: integers beyond 2^53 and long decimals are written as text; NaN and infinities become empty cells; time-zoned instants are written as UTC wall-clock times. — to ods, xls, xlsx (42 paths)
  • Date-times are stored to the millisecond. — to xls, xlsx (28 paths)
  • At most 65,536 rows and 256 columns per sheet (legacy format limit); produced through the bundled LibreOffice. — to xls (14 paths)
  • At most 1,048,576 rows and 16,384 columns per sheet (larger tables stop with an error, or are split across sheets on request). — to ods, xlsx (28 paths)
  • One sheet per output file: the first one by default, a chosen one, or all of them as separate files. — from ods, xls, xlsx (30 paths)
  • Text formats carry no types: dates are written in ISO 8601, binary values as Base64, nested values as JSON text. — to csv, tsv (30 paths)
  • JSON has no date, binary, NaN or infinity: dates become ISO 8601 text, binary values Base64 text, NaN and infinities null. — to json, jsonl (25 paths)
  • Columns are written as int64, float64, bool, utf8, binary, date32, time64[us], timestamp[us] and decimal128; nested values as JSON text. Buffers are not compressed. — to arrow (14 paths)
  • SQLite numbers are 64-bit: exact decimals and 128-bit integers are stored as TEXT; dates and times as ISO 8601 text in DATE / DATETIME / TIME columns. — to sqlite (14 paths)
  • XML carries text only; empty values are omitted, so empty text and missing values cannot be told apart when read back. — to xml (10 paths)
  • Binary values are written as Base64 text. — to yaml (10 paths)
  • TOML has no null: empty values are omitted (or stop the conversion, as chosen). — to toml (10 paths)
  • Property lists have no null (empty values are omitted or stop the conversion) and are built in memory (20 million cells at most). — to plist (10 paths)
  • Column types are detected from the text (verified on every row); numbers are re-written in canonical form (1.50 → 1.5). Identifiers with leading zeros stay text. Turn detection off to keep every cell as written. — from csv, tsv, xml (40 paths)
  • Same-format conversion: the file is re-written with the chosen options. — csv→csv, tsv→tsv, jsonl→jsonl, parquet→parquet
  • Integers beyond 64 bits are read as 64-bit floats by the JSON reader. Text that looks like an ISO date becomes a date. — from json, jsonl (25 paths)
  • The table is built from the document's list of records (detected, or given by “Record list”); other content of the document is not converted. — from json, plist, toml, xml, yaml (50 paths)
  • XML carries text only: numbers and booleans become text, empty strings and nulls both become empty elements. Keys starting with “@” become attributes, “#text” becomes element text, lists become repeated elements. — json→xml, xml→xml, yaml→xml, toml→xml, plist→xml
  • TOML has no null (nulls are omitted or stop the conversion, as chosen) and needs a table at the top level; plain values are moved before sub-tables. — json→toml, xml→toml, yaml→toml, toml→toml, plist→toml
  • Property lists have no null and no key order (keys are sorted when read back); integers beyond 64 bits and exact decimals are written as text. — json→plist, xml→plist, yaml→plist, toml→plist, plist→plist
  • ZSTD-compressed Arrow buffers, union, run-end-encoded and view types are not supported. Nested columns are read as JSON text; unsigned 64-bit integers become exact decimals. — from arrow (14 paths)
  • Only tables are read. Views, indexes, constraints and triggers are not carried over; stored SQL is never executed. — from duckdb, sqlite (28 paths)
  • One table per output file: the first one by default, a chosen one, or all of them as separate files. — from duckdb, sqlite (20 paths)
  • Dates and binary values become text (ISO 8601 / Base64); JSON has no such types. — xml→json, yaml→json, toml→json, plist→json
  • XML comments and processing instructions are dropped; attribute order is not kept; every value is read as text. Documents declaring entities (DTD) are refused. — xml→json, xml→xml, xml→yaml, xml→toml, xml→plist
  • The document is re-serialised: comments and original formatting are not kept. — xml→xml, yaml→yaml, toml→toml
  • Comments, anchors and formatting of the source are not kept. — xml→yaml, yaml→yaml, toml→yaml, plist→yaml
  • YAML comments, anchors and tags are not carried over. — yaml→json, yaml→xml, yaml→toml, yaml→plist
  • TOML comments are not carried over. — toml→json, toml→xml, toml→yaml, toml→toml, toml→plist

# 3. Type mapping

Internal column kinds: string, integer (64-bit), decimal (exact digits), double, boolean, date, timestamp (no zone), timestampTZ (UTC instant), time, binary, json (nested value). Dates/times are integers (days, microseconds): no calendar or time-zone arithmetic is applied, so nothing is shifted.

# Reading

Source integer decimal double boolean date / timestamp text other
CSV / TSV / XML records text -?(0|[1-9][0-9]*) in 64-bit range → BIGINT; wider → DECIMAL(38,0) > 15 significant digits → DECIMAL(38,s) other numbers (optional decimal comma / thousands separator) true/false (any case) ISO 8601 date, date-time (→ TIMESTAMP), date-time with offset (→ TIMESTAMPTZ, instant kept), HH:MM:SS → TIME; d/m/y or m/d/y only when unambiguous or when a format is given everything else, incl. leading-zero identifiers, +33, mixed columns detection is verified on every row; one non-conforming value keeps the column text
JSON / JSON Lines BIGINT (UBIGINT/HUGEINT when needed) — DOUBLE BOOLEAN ISO date/time strings → DATE/TIMESTAMP/TIMESTAMPTZ/TIME (verified on every row) VARCHAR objects → STRUCT, arrays → LIST (kept, flattened or JSON text, see §5); integers beyond 64 bits → DOUBLE (DuckDB reader)
YAML / TOML / plist / JSON object as JSON; integers beyond 64 bits and long decimals are carried as text (exact) native dates of the format plist Data, YAML !!binary → Base64 text
Parquet / DuckDB native DuckDB types, unchanged nested types kept / flattened / JSON text
XLSX / XLS number cell whose text is plain digits — other number cells t="b" number + date/time number format (built-in ids and custom codes), 1900 and 1904 systems; t="d" ISO cells shared, inline, formula-string cells errors → NULL (warned); mixed column → text
ODS float/percentage/currency with integer text — other floats boolean date (date or date-time), time (duration < 24 h) string mixed column → text
SQLite INTEGER storage — REAL INTEGER 0/1 in a column declared BOOL* TEXT ISO values in columns declared DATE*/TIME* TEXT BLOB → binary; a column mixing storage classes → text
Arrow int8…int64, uint8…uint32 → BIGINT; uint64 → UBIGINT decimal128 → DECIMAL(p,s) float16/32/64 bool date32/64, time32/64, timestamp (s/ms/µs/ns; ns truncated to µs with a warning; zoned → TIMESTAMPTZ) utf8, large_utf8, dictionary-encoded strings binary, large/fixed-size binary; list/struct/map/fixed-size-list → JSON text; duration → BIGINT

# Writing

Target integer decimal / 128-bit double boolean date timestamp timestampTZ time binary nested NULL vs ""
CSV / TSV digits exact digits shortest round-trip true/false ISO (or strftime option) YYYY-MM-DDTHH:MM:SS[.ffffff] same + Z ISO Base64 JSON text NULL = empty field (or chosen text), "" = ""
JSON / JSON Lines number number, digit for digit number; NaN/±inf → null (counted warning) true/false ISO string ISO string ISO string + Z string Base64 string real JSON null vs ""
Parquet INT64 etc. (native) DECIMAL; 128-bit ints → DECIMAL(38,0), or text if 39 digits DOUBLE/FLOAT BOOLEAN DATE TIMESTAMP (µs) TIMESTAMP (UTC) TIME BYTE_ARRAY native STRUCT/LIST/MAP native
DuckDB native native native native native native native native BLOB native native
Arrow int64 decimal128(p,s); 128-bit ints as Parquet float64 bool date32 timestamp[us] timestamp[us, UTC] time64[us] binary utf8 (JSON text) validity bitmap
SQLite INTEGER TEXT (exact) REAL (NaN → text nan) BOOLEAN 0/1 DATE ISO text DATETIME ISO text DATETIME ISO text + Z TIME text BLOB TEXT (JSON) NULL vs ''
XLSX / XLS number; beyond ±2^53 → text (option: number) ≤ 15 digits → number, else text number; NaN/±inf → empty (counted) boolean date cell (yyyy-mm-dd), outside 1900-03-01…9999 → ISO text date-time cell, millisecond precision (finer → rounded, counted) UTC wall clock, zone dropped (warned) time cell Base64 text JSON text empty cell vs empty string cell (XLS: both empty)
ODS float as XLSX float boolean date date with full microseconds UTC wall clock (warned) time duration Base64 text JSON text empty cell vs empty string cell
XML (records) text text text text text text text text Base64 JSON text → text NULL = element omitted; "" reads back as NULL
YAML (list of mappings) int float literal, exact digits float (.nan, .inf) bool timestamp timestamp timestamp Z quoted string Base64 string real nested YAML null vs ""
TOML ([[rows]]) integer float literal / string for > 64-bit integers float (nan, inf) boolean local date local date-time offset date-time local time Base64 string inline tables / arrays NULL omitted (or error)
plist (array of dicts) integer string (exact) real boolean string string Date (XML plists: to the second, warned) string Data nested dict/array NULL omitted (or error)

# 4. Reading and writing options (all exposed as OptionDescriptors)

Reading: encoding.input, csv.delimiter.input, csv.quote.input, csv.lenient, nulls.input, header (auto/yes/no), header.row, types.detect (off = all text), types.columns (name=text|integer|decimal|boolean| date|timestamp), numbers.decimal.comma, numbers.thousands, dates.input.format, sheet (name or #n), sheets.all, formulas.as.text, records.path, nested (auto/flatten/json/native), columns.select, columns.rename. Writing: csv.preset (standard / Excel-friendly = UTF-8 BOM + CRLF), csv.delimiter, csv.header, csv.quote.all, csv.line.ending, encoding.output, encoding.bom, nulls.output, dates.output.format, formula.guard (CSV-injection guard, off by default: prefixes text cells starting with = + - @ with ', counted and reported), json.pretty, xml.root.name, xml.row.name, xml.item.name, nulls.policy (omit/error, TOML and plist), records.key, plist.format, parquet.compression (zstd/snappy/gzip/lz4/uncompressed), parquet.row.group.size, arrow.format (file/stream), sheet.header.bold, sheet.header.freeze, sheet.overflow (error/split), integers.big, xlsx.strings, table.name, memory.limit.mb, threads.

  • Encoding detection: BOM (UTF-8, UTF-16 LE/BE) → UTF-8 validity → UTF-16 without BOM (NUL pattern) → single-byte heuristic (no 0x80–0x9F bytes: ISO-8859-1; bytes undefined in Windows-1252 or accented letters inside words: Mac Roman; otherwise Windows-1252). A heuristic answer raises the encoding-guessed warning. Non-UTF-8 input is transcoded to UTF-8 in the workspace before DuckDB reads it.
  • Delimiter: engine sniffer over comma, semicolon, tab, pipe (quote-aware, first 64 KB); quote/escape/header: DuckDB's sniffer for that delimiter. Rows of unequal length stop the conversion with an explanation unless csv.lenient is on.
  • NULL on input: an unquoted empty field is NULL, "" is an empty string; extra NULL tokens via nulls.input apply to unquoted fields only.
  • Sheets/tables: to a workbook or database target, every sheet/table is carried by default; to a single-table target, the first visible one (others listed by name in a sheets-dropped data-loss warning) or all of them as <name>-<sheet>.<ext> files.

# 5. Nested values (JSON, Parquet, DuckDB, Arrow, YAML/TOML/plist/XML records)

nested effect
flatten (default for flat targets) each STRUCT field becomes a column parent.child (recursively); lists and maps become JSON text
json every nested column becomes one JSON-text column
native (default for Parquet, DuckDB, JSON, JSON Lines targets) nested types are kept as they are

Arrow nested columns are always read as JSON text. A JSON-text column written to JSON/YAML/TOML/plist is re-expanded into real nested values.

# 6. No silent loss — warnings

preflight opens the real file exactly as convert would (first 20 000 rows per table; counts taken from a partial read say so) and adds the target's schema-level effects; convert raises the same warnings with exact counts. Codes: sheets-dropped (names), formulas-dropped / formulas-as-text (count), error-cells, merged-cells, hidden-rows-columns, hidden-sheet, sheet-features-dropped (data validation, conditional formatting, hyperlinks), workbook-content-dropped (charts, images, pivot tables, comments, external links, macros, defined names — detected from the package parts), blank-rows-skipped, mixed-types, columns-renamed, columns-not-selected, encoding-guessed, undecodable-characters, unencodable-characters, ambiguous-date-format, type-detection-reverted, timezone-normalized, timezone-dropped, sub-millisecond-dropped, sub-microsecond-dropped, plist-subsecond-dropped, integer-as-text, decimal-as-text, int128-as-decimal, int128-as-text, wide-number-as-text, non-finite-dropped, date-as-text, time-as-text, binary-as-text, cell-too-long, csv-injection-guarded, nested-flattened, nested-stringified, json-number-precision, spreadsheet-number-precision, spreadsheet-time-precision, content-outside-records, records-located, single-record, duplicate-keys, non-string-keys, yaml-multiple-documents, xml-comments-dropped, xml-processing-instructions-dropped, xml-mixed-content, xml-sibling-order, xml-names-adapted, xml-empty-strings, xml-types-as-text, toml-root-wrapped, key-order-changed, null-omitted, control-characters-replaced, views-not-converted, sheets-renamed, tables-renamed, sheet-split, xls-via-libreoffice, xls-column-limit.

validate re-opens the output in a fresh locked-down session and checks: it opens; number of tables; exact row count read back vs rows written; column count; column names; and the cells of the first 1 000 rows of each table against an independent read of the source (numbers compared by value, date-times by instant). On lossless paths a single differing cell fails validation; on the others the comparison is reported (test validationCatchesAWrongOutput). Tree paths: the output is parsed back; lossless paths must be identical, others report the comparison of all scalar values.

inspect returns: sheets/tables with rows × columns (workbooks are scanned up to 96 MB of compressed sheet data; beyond that counts are "at least"), formulas, merged ranges, hidden rows/columns, charts/images/pivots/macros; text encoding, delimiter, quote, header, detected column types; Parquet created_by, row groups, format version, compression codecs and physical/logical schema; SQLite page size, page count, user_version, application_id, encoding, journal mode, tables/views with row counts and declared column types; DuckDB table list with row counts and schema (and the DuckDB version used to read it); Arrow variant, row count and Arrow types; for documents, the top-level shape, the detected record list and the reader's notes.

# 7. Security limits

Limit Value Behaviour
ZIP entries 200 000 limitExceeded
Small package part (workbook, styles, rels) 256 MB decompressed limitExceeded
Streamed part (worksheet, content.xml, shared strings) 16 GB decompressed, counted on bytes actually produced, never on declared sizes; CRC-32 verified limitExceeded / malformedInput
Shared strings in memory 1 GB limitExceeded
Sheet 1 048 576 rows × 16 384 columns limitExceeded
ODS repeated rows/cells materialised cells ≤ 500 million; trailing empty repeats cost nothing; a repeat that cannot fit is refused before any row is produced limitExceeded
XML token (package parts) 128 MB limitExceeded
DOCTYPE / entities refused everywhere: package parts (tokenizer has no DTD support) and XML documents (byte pre-check + parser delegate; shouldResolveExternalEntities = false) malformedInput
Tree documents 512 MB file, 256 levels, 50 million values; YAML alias expansion bounded by input size limitExceeded
SQLite read-only, defensive mode, trusted_schema off, views/triggers disabled, no ATTACH, value ≤ 512 MB; only tables are read error
Arrow message header ≤ 256 MB, decompressed buffer ≤ 2 GB, every offset/length checked error
Encrypted input OLE-wrapped (password-protected) XLSX, encrypted ZIP entries, encrypted ODS encryptedInput
Macros never executed (VBA parts are only counted) —

# 8. Performance (measured)

Release build (swift test -c release -Xswiftc -enable-testing --filter Benchmarks, harness Tests/EngineDataTests/Benchmarks.swift), one process per benchmark so that the peak resident size (ru_maxrss) is that benchmark's. Inputs generated by DuckDB: 9 columns (integer id, 32-char token, text, decimal amount, boolean, date, timestamp, category, zero-padded zip). Times include type detection + its full-file verification pass and exclude validate. Defaults: memory limit 4 096 MB, 8 threads, unless stated.

The Mac was shared with other heavy builds and test runs during these measurements (1-minute load average between 23 and 84 on 18 cores). The numbers are what was observed, not best cases; they vary from run to run.

Conversion Rows Source Output Time Rows/s Source MB/s Peak RSS validate Load avg (1 min)
CSV (2.36 GB) → Parquet, zstd 22 000 000 2 363 MB 538 MB 6.90 s 3 187 274 342 1 558 MB 0.29 s, all passed 83.84
CSV (2.36 GB) → Parquet, snappy 22 000 000 2 363 MB 1 226 MB 4.57 s 4 818 602 518 1 403 MB 0.30 s, all passed 45.57
CSV (2.36 GB) → Parquet, zstd, memory limit 1 024 MB 22 000 000 2 363 MB 538 MB 8.25 s 2 666 335 286 1 210 MB 0.24 s, all passed 44.72
CSV (2.36 GB) → Parquet, zstd, memory limit 512 MB, 4 threads 22 000 000 2 363 MB 538 MB 7.49 s 2 936 470 315 812 MB 0.16 s, all passed 44.74
CSV (2.36 GB) → Parquet, zstd, type detection off 22 000 000 2 363 MB 462 MB 2.43 s 9 067 708 974 1 961 MB 0.17 s, all passed 40.77
Parquet (zstd) → CSV 22 000 000 538 MB 2 363 MB 3.10 s 7 096 124 174 2 071 MB 1.13 s, all passed 40.77
Parquet (zstd) → CSV, memory limit 512 MB, 4 threads 22 000 000 538 MB 2 363 MB 4.17 s 5 276 365 129 758 MB 1.35 s, all passed 46.02
CSV → XLSX (1 M rows, shared strings) 1 000 000 105 MB 73 MB 6.16 s 162 428 17 384 MB 4.66 s, all passed 45.38
CSV → XLSX (1 M rows, inline strings) 1 000 000 105 MB 79 MB 4.44 s 225 321 24 207 MB 3.87 s, all passed 45.43
XLSX (1 M rows) → CSV 1 000 000 73 MB 105 MB 4.55 s 219 933 16 552 MB 1.00 s, all passed 39.44
CSV → ODS (1 M rows) 1 000 000 105 MB 73 MB 5.43 s 184 246 20 206 MB 8.83 s, all passed 37.65
JSON Lines (1.45 GB) → Parquet, zstd 8 000 000 1 480 MB 195 MB 12.99 s 615 644 114 3 685 MB 0.46 s, all passed 31.53
CSV (2.36 GB) → SQLite 22 000 000 2 363 MB 2 431 MB 33.07 s 665 267 71 503 MB 0.66 s, all passed 26.86
CSV (2.36 GB) → Arrow IPC file 22 000 000 2 363 MB 2 094 MB 13.43 s 1 637 648 176 443 MB 0.20 s, all passed 22.64

Reading the table: CSV → Parquet reads the 2.36 GB file twice (verification of detected types, then the copy); with detection off it is read once. With a 512 MB limit the same 2.36 GB conversions complete in well under the file's size in resident memory, which is the larger-than-memory proof asked for. JSON Lines → Parquet detects the schema on the whole file before converting; its peak of 3.7 GB was measured with the default 4 GB limit and was not measured with a lower one. CSV → SQLite is bound by single-threaded row inserts. XLSX → CSV keeps the workbook's shared strings in memory (1 M distinct 32-character tokens here).

# 9. Evaluated and NOT implemented

  • Apache arrow-swift (evaluated at commit 15dfba80, 2026-11-10): it builds with Swift 6.4.2 (tools 5.10, Swift 5 language mode) but pulls 21 transitive packages (grpc-swift, swift-nio, swift-protobuf… for Arrow Flight), and its reader failed on files written by Arrow C++ (pyarrow 24): Unsupported field type found: largeutf8, and …: decimal on a file with a decimal128 column. Rejected; the engine has its own reader/writer, cross-checked during development against pyarrow 24 in both directions (pyarrow validate(full=True) on our files); the tests use four small files written by pyarrow (Tests/EngineDataTests/Fixtures) and a byte-level layout test, and do not need Python.
  • Arrow: ZSTD-compressed buffers (no public Zstandard decoder on macOS; clear error telling to re-save without ZSTD), union, run-end-encoded, string-view/binary-view, list-view, interval, decimal256, big-endian files, dictionary-encoded or nested columns on write, buffer compression on write, Feather v1.
  • DuckDB extensions (sqlite, excel, arrow, spatial, httpfs…): not used by design (no download, no network).
  • Native BIFF8 (XLS) parser: not written; XLS goes through the bundled LibreOffice (a soffice process is started for each conversion; its cost was not benchmarked). Without LibreOffice in the build, XLS paths are not declared.
  • XLSX writer: formulas, styles beyond the bold frozen header and date formats, column widths, charts, multiple header rows, ZIP64 (a package part over 4 GB stops with limitExceeded). XLSX reader: formulas are not evaluated (cached values are read); rich-text formatting, comments and hyperlinks are dropped (reported); XLSB and password-protected workbooks are not read.
  • ODS: freeze panes on write; formula evaluation; sub-tables.
  • Views in SQLite and DuckDB files: listed, never executed. Indexes, constraints and triggers are not carried between databases. SQLite → SQLite and DuckDB → DuckDB are not offered (copy the file).
  • XML: DTDs/entities are refused (also harmless ones); namespaces are kept as written prefixes, not resolved; mixed content keeps its text but not its position; documents over 512 MB (tree in memory).
  • JSON tabular input: integers beyond 64 bits become doubles (DuckDB reader) for top-level arrays and JSON Lines; exact handling only for documents read by the engine's parser (JSON objects, YAML, TOML, plist, where such numbers are carried as text).
  • YAML: comments, anchors, custom tags and block styles are not preserved; !!binary is written as a Base64 string (Yams' emitter does not write tags). TOML: comments not preserved. plist: key order (Apple's API).
  • Text encodings other than UTF-8, UTF-16 LE/BE, Windows-1252, ISO-8859-1, Mac Roman.
  • Geospatial and scientific formats (GeoJSON, GPKG, HDF5, NetCDF…): out of scope of this engine by assignment.

# 9b. XML mapping of a document's top level (fixed 2026-11-21)

An XML document has exactly one root element. JSON, YAML and property lists may hold anything at their top level, so the tree → XML writer maps the top level as follows (Tree/XMLTree.swift, XMLTreeCodec.Writer.serialize):

Top-level value Written as Reads back (XML → JSON) as
object with one key whose value is not a list that key is the root element the same object
any other object <root> holding the members {"root": {…}}
list [a, b, …] <root><item>a</item><item>b</item>…</root> the same list (see the rule below)
list inside a list <item> holding <item> elements {"item": [...]} per inner list (XML has no anonymous list)
empty list <root/> (warning xml-empty-list) null
scalar / text <root>text</root> {"root": "text"}
null <root/> {"root": null}

Reading rule that makes top-level lists round-trip: a root element named root whose only children are item elements (no attribute, no text) is read as a list of those items. Consequences, all intended: [{"id":1},{"id":2}] → XML → JSON gives a list of two records again (values as text, as for every XML path); a one-entry list stays a list; an object written by hand as {"item": [...]} also reads back as a list. The element names can be changed (xml.root.name, xml.item.name); with other names the list is found under <root name>.<item name> when read back, and the xml-root-wrapped warning says so. An empty list under a key is written as one empty element and reported (xml-empty-list) instead of vanishing.

Before this fix a top-level list of two or more entries was written as several root elements (<root>…</root> <root>…</root>): not XML. The engine's own validation caught it, so json→xml, yaml→xml and plist→xml failed on every such document. Tests: TreeTests.topLevelValuesToXML (27 cases: nine shapes × JSON, YAML, plist), TreeTests.topLevelListRoundTripsThroughXML.

# 10. Known defects and caveats

  • The official DuckDB library is a 62 MB dynamic library declaring macOS 11.0 as its minimum; it must be embedded and signed with the app (Contents/Frameworks).
  • An unterminated quote in a CSV is accepted by DuckDB (the field runs to the end of the file) rather than rejected.
  • A zero-byte file with a SQLite extension is a valid empty database for SQLite: it inspects fine and convert answers "contains no ordinary table".
  • XLSX output with shared strings keeps up to 1 000 000 distinct strings / 256 MB in memory, then switches to inline strings; reading a workbook loads its shared-string table in memory (1 GB cap).
  • validate reads the output again: for large XLSX/ODS outputs this costs about as much as the conversion.
  • Preflight on very large documents (> 64 MB tree formats) is skipped and says so.
  • Tree → XML: a list nested directly in a list, an empty list and a top-level scalar do not read back as what they were (see §9b); an object whose only key is item under a root named root reads back as a list.
  • DataTests.cancellationStopsMidStream (CSV → Parquet case) is timing-sensitive: on a heavily loaded Mac (load average above 40 on 18 cores) the conversion once finished before the cancellation was seen and the test failed; it passed when re-run alone. The test, not the engine, depends on the load.

# 11. Tests

swift test from Packages/EngineData (debug), 2026-11-21, with the bundled LibreOffice present: Test run with 70 tests in 6 suites passed after 55.780 seconds — 67 tests run (297 parameterised cases, among them the 194 tabular and 25 tree paths and the 27 top-level XML shapes of §9b), 3 benchmark entries disabled unless ORIX_DATA_BENCH is set.

Suite What it covers
MatrixTests every declared tabular path: convert the canonical dataset (unicode incl. emoji and RTL, quotes, delimiters, line breaks, integers beyond 2^53, decimals, booleans, dates, timestamps, NULL vs empty string, leading-zero identifiers), run validate, read back and compare every cell
TreeTests every declared tree path; JSON exactness (key order, big integers, decimal text, escapes, surrogates), YAML typing and quoting, TOML 1.0 features and rejections, XML mapping convention and its reversibility, XXE (file:// entity, parameter entity, UTF-16), billion laughs, YAML alias bomb, 100 000-level nesting, null policies, malformed documents
SpreadsheetTests hand-assembled Excel-style packages (shared/inline/rich strings, phonetic runs, _xHHHH_ escapes, built-in and custom date formats, 1900 and 1904 systems, cached formula values, shared formulas, errors, merged cells, hidden sheets/rows/columns, prefixed elements, charts/images/pivots/comments/macros parts), number-format classification, writers' edge cases and loss tallies, sheet overflow (error and split), ODS typed cells and repeats, repeat bombs, ZIP reader (bomb limit, truncation, checksum, unzip -t), tokenizer, LibreOffice cross-check of our XLSX and ODS (conditional), XLS limits (conditional)
TextTests delimiter/quote/header sniffing (semicolon + decimal comma, tab, pipe, quoted line breaks), encodings (UTF-8 ± BOM, UTF-16 LE/BE ± BOM, Latin-1, Windows-1252, Mac Roman, forced), conservative type detection and its full-file verification, forced types, NULL tokens, skipped rows, ragged rows, writing options, Excel preset, CSV-injection guard, column selection/renaming
DataTests declared contract (path count, options, defaults), Arrow files written by Arrow C++ (plain, LZ4, stream), ZSTD refusal, damaged Arrow, byte-level IPC layout of our files, SQLite (types, views, triggers, hostile names, read-only hardening), DuckDB files (all types through every streamed target), Parquet codecs/row groups/metadata and broken footers, nested JSON (flatten/json/native, record paths), damaged inputs for every reader, hostile file/table/column names, the DuckDB sandbox, memory limit and spilling, cancellation mid-stream for five targets, preview API, validation catching wrong outputs
Files main
Menu

Theme