Data engine (data) — spreadsheets, tables and structured documents
Package Packages/EngineData (module EngineData, factory makeEngines()), engine id data, display name
"Spreadsheets & Data". Everything below was implemented, run and measured on 2026-11-21 on this Mac
(Apple M-series, 18 cores, 48 GB, macOS 27.0, Swift 6.4.2). Nothing is listed that is not covered by a passing test.
Formats: XLSX, XLS, ODS, CSV, TSV, JSON, JSON Lines, Parquet, Arrow IPC (file = Feather v2, and stream), SQLite, DuckDB database files, XML, YAML, TOML, property lists (XML and binary).
1. Architecture in one page
- DuckDB v2.0.1 (official
libduckdb, arm64, C API) is the tabular hub. Every tabular conversion is "source → DuckDB relation → target":- CSV/TSV, JSON/JSON Lines, Parquet and DuckDB files are scanned by DuckDB itself (streaming, multi-threaded,
bounded by
memory_limit, spilling to atemp_directoryinside the job's work directory). - XLSX, ODS, XLS, SQLite and Arrow are read by the engine's own readers and staged, row by row, into a DuckDB table that lives in a database file inside the work directory (so they are not limited by memory either). Each column records the kinds of values actually seen and is cast to the narrowest type that holds them all exactly.
- XML, YAML, TOML, property lists and JSON objects are parsed into an ordered value tree; the list of records is located and handed to DuckDB.
- CSV/TSV/JSON/JSON Lines/Parquet are written by DuckDB
COPY, DuckDB files byCREATE TABLE AS; everything else is written by the engine's own writers from streamed result chunks.
- CSV/TSV, JSON/JSON Lines, Parquet and DuckDB files are scanned by DuckDB itself (streaming, multi-threaded,
bounded by
- Tree conversions (JSON, YAML, TOML, XML, plist ↔ each other) never go through a table: parse → ordered tree → serialise.
- Sandbox. Each job opens its own DuckDB session with
autoinstall_known_extensions=false,autoload_known_extensions=false,enable_external_access=false,allowed_directories= the job's private workspace + the work directory,allowed_paths= the job's source file,TimeZone='UTC', thenlock_configuration=true. SQL can therefore not read or write any other file, cannot install or load an extension and cannot undo these settings (testduckdbIsLockedDown: 23 escape attempts, all refused). No user text is ever spliced into SQL unquoted: names and paths go throughSQL.ident/SQL.literal(testhostileFileSheetAndColumnNames). A source file whose name contains glob characters (* ? [ ] { }) is reached through a hard link with a neutral name, so the name can never match another file. - What the bundled DuckDB build contains (verified by running it): core functions, CSV,
parquet,json,icu. Its SQLite, Excel, Arrow and spatial support live in downloadable extensions and are not used. - The pipeline is synchronous and runs on a dedicated 8 MB-stack thread per job; task cancellation interrupts the running DuckDB statement / SQLite statement / LibreOffice process and is checked in every row loop.
- Progress is measured: DuckDB's own scan progress for
COPY/verification passes, compressed bytes consumed for XLSX/ODS, rows written over rows staged for streamed targets, file offset for Arrow. A job declares how many full passes over the data it makes (1 or 2) and reports (finished passes + fraction of the current one) / passes. Steps that cannot be measured (LibreOffice, tree documents) reportfraction: nil.
Bundled / used components
| Component | Version | License | How | Size |
|---|---|---|---|---|
DuckDB libduckdb.dylib |
v2.0.1 (pinned, SHA-256 in EmbeddedEngines/duckdb/build.sh) |
MIT | dynamic library in CDuckDB.xcframework, SwiftPM binaryTarget |
62 MB (arm64) |
| Yams | 6.2.2 (exact pin) | MIT | SwiftPM source dependency (wraps libyaml, compiled by SwiftPM) | source |
| SQLite | system libsqlite3 (3.54.0 on this Mac) |
public domain | linked from the macOS SDK | — |
Compression, zlib (crc32 only) |
system | — | Apple frameworks / SDK | — |
LibreOffice (soffice) |
bundled by the documents engine in EmbeddedEngines/tools/LibreOffice.app |
MPL-2.0 | separate process through ProcessRunner, XLS only |
not owned by this engine |
Written for this engine (no third-party code): ZIP reader/writer, streaming XML tokenizer, XLSX reader/writer, ODS reader/writer, Arrow IPC reader/writer (FlatBuffers metadata + LZ4-frame decoder), exact JSON parser/serialiser, TOML 1.0 parser/serialiser, XML ↔ tree mapping, text-encoding detection/transcoding, delimiter sniffer, column type detection.
2. Declared paths
219 paths are declared when the bundled LibreOffice is present (as today), 191 otherwise: the 28 paths that
read or write XLS exist only with it (paths is computed; convert answers engineUnavailable if XLS is requested
without it). No named operations; every path is a plain format conversion. Same-format paths exist only where
re-writing with other options is meaningful and safe: CSV, TSV, JSON Lines, Parquet, JSON, YAML, TOML, XML, plist.
Status is supported for all of them: each one converts a fixture, passes the engine's own validate, and is read
back and compared value by value in the parameterised tests MatrixTests.path (194 tabular paths) and
TreeTests.path (25 tree paths).
Fidelity as declared
lossless: Parquet → Parquet, Parquet → DuckDB, JSON → JSON, JSON → YAML, plist → plist. The output reads back identical (types and values; for plist, the same property list).partial: every path from a workbook (XLSX/XLS/ODS: only cell values are carried), and every path from a database file to a single-table target (one table per output file).structural: everything else — the content is kept but re-expressed (types detected from text, dates as ISO text, nested values flattened or stringified…). On these paths the canonical dataset of the tests still reads back with equal values (compared as numbers / instants / text); the differences are type re-expression, documented below.
Full path table (219 rows)
| Source | Destination | Operation | Status | Fidelity | Preserves | Backend (read → write) | Test |
|---|---|---|---|---|---|---|---|
| xlsx | xls | — | supported | partial | multipleSheets, tables | engine XLSX reader → staging → engine XLSX writer → LibreOffice | MatrixTests.path (xlsx → xls) |
| xlsx | ods | — | supported | partial | multipleSheets, tables | engine XLSX reader → staging → engine ODS writer | MatrixTests.path (xlsx → ods) |
| xlsx | csv | — | supported | partial | tables | engine XLSX reader → staging → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (xlsx → csv) |
| xlsx | tsv | — | supported | partial | tables | engine XLSX reader → staging → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (xlsx → tsv) |
| xlsx | json | — | supported | partial | tables | engine XLSX reader → staging → DuckDB COPY … (FORMAT json) |
MatrixTests.path (xlsx → json) |
| xlsx | jsonl | — | supported | partial | tables | engine XLSX reader → staging → DuckDB COPY … (FORMAT json) |
MatrixTests.path (xlsx → jsonl) |
| xlsx | parquet | — | supported | partial | tables | engine XLSX reader → staging → DuckDB COPY … (FORMAT parquet) |
MatrixTests.path (xlsx → parquet) |
| xlsx | arrow | — | supported | partial | tables | engine XLSX reader → staging → engine Arrow IPC writer | MatrixTests.path (xlsx → arrow) |
| xlsx | sqlite | — | supported | partial | multipleSheets, tables | engine XLSX reader → staging → system libsqlite3 | MatrixTests.path (xlsx → sqlite) |
| xlsx | duckdb | — | supported | partial | multipleSheets, tables | engine XLSX reader → staging → DuckDB CREATE TABLE AS |
MatrixTests.path (xlsx → duckdb) |
| xlsx | xml | — | supported | partial | tables | engine XLSX reader → staging → engine XML writer | MatrixTests.path (xlsx → xml) |
| xlsx | yaml | — | supported | partial | tables | engine XLSX reader → staging → Yams (row by row) | MatrixTests.path (xlsx → yaml) |
| xlsx | toml | — | supported | partial | tables | engine XLSX reader → staging → engine TOML writer | MatrixTests.path (xlsx → toml) |
| xlsx | plist | — | supported | partial | tables | engine XLSX reader → staging → PropertyListSerialization | MatrixTests.path (xlsx → plist) |
| xls | xlsx | — | supported | partial | multipleSheets, tables | LibreOffice → XLSX → engine XLSX reader → engine XLSX writer | MatrixTests.path (xls → xlsx) |
| xls | ods | — | supported | partial | multipleSheets, tables | LibreOffice → XLSX → engine XLSX reader → engine ODS writer | MatrixTests.path (xls → ods) |
| xls | csv | — | supported | partial | tables | LibreOffice → XLSX → engine XLSX reader → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (xls → csv) |
| xls | tsv | — | supported | partial | tables | LibreOffice → XLSX → engine XLSX reader → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (xls → tsv) |
| xls | json | — | supported | partial | tables | LibreOffice → XLSX → engine XLSX reader → DuckDB COPY … (FORMAT json) |
MatrixTests.path (xls → json) |
| xls | jsonl | — | supported | partial | tables | LibreOffice → XLSX → engine XLSX reader → DuckDB COPY … (FORMAT json) |
MatrixTests.path (xls → jsonl) |
| xls | parquet | — | supported | partial | tables | LibreOffice → XLSX → engine XLSX reader → DuckDB COPY … (FORMAT parquet) |
MatrixTests.path (xls → parquet) |
| xls | arrow | — | supported | partial | tables | LibreOffice → XLSX → engine XLSX reader → engine Arrow IPC writer | MatrixTests.path (xls → arrow) |
| xls | sqlite | — | supported | partial | multipleSheets, tables | LibreOffice → XLSX → engine XLSX reader → system libsqlite3 | MatrixTests.path (xls → sqlite) |
| xls | duckdb | — | supported | partial | multipleSheets, tables | LibreOffice → XLSX → engine XLSX reader → DuckDB CREATE TABLE AS |
MatrixTests.path (xls → duckdb) |
| xls | xml | — | supported | partial | tables | LibreOffice → XLSX → engine XLSX reader → engine XML writer | MatrixTests.path (xls → xml) |
| xls | yaml | — | supported | partial | tables | LibreOffice → XLSX → engine XLSX reader → Yams (row by row) | MatrixTests.path (xls → yaml) |
| xls | toml | — | supported | partial | tables | LibreOffice → XLSX → engine XLSX reader → engine TOML writer | MatrixTests.path (xls → toml) |
| xls | plist | — | supported | partial | tables | LibreOffice → XLSX → engine XLSX reader → PropertyListSerialization | MatrixTests.path (xls → plist) |
| ods | xlsx | — | supported | partial | multipleSheets, tables | engine ODS reader → staging → engine XLSX writer | MatrixTests.path (ods → xlsx) |
| ods | xls | — | supported | partial | multipleSheets, tables | engine ODS reader → staging → engine XLSX writer → LibreOffice | MatrixTests.path (ods → xls) |
| ods | csv | — | supported | partial | tables | engine ODS reader → staging → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (ods → csv) |
| ods | tsv | — | supported | partial | tables | engine ODS reader → staging → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (ods → tsv) |
| ods | json | — | supported | partial | tables | engine ODS reader → staging → DuckDB COPY … (FORMAT json) |
MatrixTests.path (ods → json) |
| ods | jsonl | — | supported | partial | tables | engine ODS reader → staging → DuckDB COPY … (FORMAT json) |
MatrixTests.path (ods → jsonl) |
| ods | parquet | — | supported | partial | tables | engine ODS reader → staging → DuckDB COPY … (FORMAT parquet) |
MatrixTests.path (ods → parquet) |
| ods | arrow | — | supported | partial | tables | engine ODS reader → staging → engine Arrow IPC writer | MatrixTests.path (ods → arrow) |
| ods | sqlite | — | supported | partial | multipleSheets, tables | engine ODS reader → staging → system libsqlite3 | MatrixTests.path (ods → sqlite) |
| ods | duckdb | — | supported | partial | multipleSheets, tables | engine ODS reader → staging → DuckDB CREATE TABLE AS |
MatrixTests.path (ods → duckdb) |
| ods | xml | — | supported | partial | tables | engine ODS reader → staging → engine XML writer | MatrixTests.path (ods → xml) |
| ods | yaml | — | supported | partial | tables | engine ODS reader → staging → Yams (row by row) | MatrixTests.path (ods → yaml) |
| ods | toml | — | supported | partial | tables | engine ODS reader → staging → engine TOML writer | MatrixTests.path (ods → toml) |
| ods | plist | — | supported | partial | tables | engine ODS reader → staging → PropertyListSerialization | MatrixTests.path (ods → plist) |
| csv | xlsx | — | supported | structural | tables | DuckDB read_csv + engine type detection → engine XLSX writer |
MatrixTests.path (csv → xlsx) |
| csv | xls | — | supported | structural | tables | DuckDB read_csv + engine type detection → engine XLSX writer → LibreOffice |
MatrixTests.path (csv → xls) |
| csv | ods | — | supported | structural | tables | DuckDB read_csv + engine type detection → engine ODS writer |
MatrixTests.path (csv → ods) |
| csv | csv | — | supported | structural | tables | DuckDB read_csv + engine type detection → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (csv → csv) |
| csv | tsv | — | supported | structural | tables | DuckDB read_csv + engine type detection → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (csv → tsv) |
| csv | json | — | supported | structural | tables | DuckDB read_csv + engine type detection → DuckDB COPY … (FORMAT json) |
MatrixTests.path (csv → json) |
| csv | jsonl | — | supported | structural | tables | DuckDB read_csv + engine type detection → DuckDB COPY … (FORMAT json) |
MatrixTests.path (csv → jsonl) |
| csv | parquet | — | supported | structural | tables | DuckDB read_csv + engine type detection → DuckDB COPY … (FORMAT parquet) |
MatrixTests.path (csv → parquet) |
| csv | arrow | — | supported | structural | tables | DuckDB read_csv + engine type detection → engine Arrow IPC writer |
MatrixTests.path (csv → arrow) |
| csv | sqlite | — | supported | structural | tables | DuckDB read_csv + engine type detection → system libsqlite3 |
MatrixTests.path (csv → sqlite) |
| csv | duckdb | — | supported | structural | tables | DuckDB read_csv + engine type detection → DuckDB CREATE TABLE AS |
MatrixTests.path (csv → duckdb) |
| csv | xml | — | supported | structural | tables | DuckDB read_csv + engine type detection → engine XML writer |
MatrixTests.path (csv → xml) |
| csv | yaml | — | supported | structural | tables | DuckDB read_csv + engine type detection → Yams (row by row) |
MatrixTests.path (csv → yaml) |
| csv | toml | — | supported | structural | tables | DuckDB read_csv + engine type detection → engine TOML writer |
MatrixTests.path (csv → toml) |
| csv | plist | — | supported | structural | tables | DuckDB read_csv + engine type detection → PropertyListSerialization |
MatrixTests.path (csv → plist) |
| tsv | xlsx | — | supported | structural | tables | DuckDB read_csv + engine type detection → engine XLSX writer |
MatrixTests.path (tsv → xlsx) |
| tsv | xls | — | supported | structural | tables | DuckDB read_csv + engine type detection → engine XLSX writer → LibreOffice |
MatrixTests.path (tsv → xls) |
| tsv | ods | — | supported | structural | tables | DuckDB read_csv + engine type detection → engine ODS writer |
MatrixTests.path (tsv → ods) |
| tsv | csv | — | supported | structural | tables | DuckDB read_csv + engine type detection → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (tsv → csv) |
| tsv | tsv | — | supported | structural | tables | DuckDB read_csv + engine type detection → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (tsv → tsv) |
| tsv | json | — | supported | structural | tables | DuckDB read_csv + engine type detection → DuckDB COPY … (FORMAT json) |
MatrixTests.path (tsv → json) |
| tsv | jsonl | — | supported | structural | tables | DuckDB read_csv + engine type detection → DuckDB COPY … (FORMAT json) |
MatrixTests.path (tsv → jsonl) |
| tsv | parquet | — | supported | structural | tables | DuckDB read_csv + engine type detection → DuckDB COPY … (FORMAT parquet) |
MatrixTests.path (tsv → parquet) |
| tsv | arrow | — | supported | structural | tables | DuckDB read_csv + engine type detection → engine Arrow IPC writer |
MatrixTests.path (tsv → arrow) |
| tsv | sqlite | — | supported | structural | tables | DuckDB read_csv + engine type detection → system libsqlite3 |
MatrixTests.path (tsv → sqlite) |
| tsv | duckdb | — | supported | structural | tables | DuckDB read_csv + engine type detection → DuckDB CREATE TABLE AS |
MatrixTests.path (tsv → duckdb) |
| tsv | xml | — | supported | structural | tables | DuckDB read_csv + engine type detection → engine XML writer |
MatrixTests.path (tsv → xml) |
| tsv | yaml | — | supported | structural | tables | DuckDB read_csv + engine type detection → Yams (row by row) |
MatrixTests.path (tsv → yaml) |
| tsv | toml | — | supported | structural | tables | DuckDB read_csv + engine type detection → engine TOML writer |
MatrixTests.path (tsv → toml) |
| tsv | plist | — | supported | structural | tables | DuckDB read_csv + engine type detection → PropertyListSerialization |
MatrixTests.path (tsv → plist) |
| json | xlsx | — | supported | structural | tables | DuckDB read_json (top-level array) / engine JSON parser (object) → engine XLSX writer |
MatrixTests.path (json → xlsx) |
| json | xls | — | supported | structural | tables | DuckDB read_json (top-level array) / engine JSON parser (object) → engine XLSX writer → LibreOffice |
MatrixTests.path (json → xls) |
| json | ods | — | supported | structural | tables | DuckDB read_json (top-level array) / engine JSON parser (object) → engine ODS writer |
MatrixTests.path (json → ods) |
| json | csv | — | supported | structural | tables | DuckDB read_json (top-level array) / engine JSON parser (object) → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (json → csv) |
| json | tsv | — | supported | structural | tables | DuckDB read_json (top-level array) / engine JSON parser (object) → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (json → tsv) |
| json | json | — | supported | lossless | engine JSON codec → engine JSON codec | TreeTests.path (json → json) |
|
| json | jsonl | — | supported | structural | tables | DuckDB read_json (top-level array) / engine JSON parser (object) → DuckDB COPY … (FORMAT json) |
MatrixTests.path (json → jsonl) |
| json | parquet | — | supported | structural | tables | DuckDB read_json (top-level array) / engine JSON parser (object) → DuckDB COPY … (FORMAT parquet) |
MatrixTests.path (json → parquet) |
| json | arrow | — | supported | structural | tables | DuckDB read_json (top-level array) / engine JSON parser (object) → engine Arrow IPC writer |
MatrixTests.path (json → arrow) |
| json | sqlite | — | supported | structural | tables | DuckDB read_json (top-level array) / engine JSON parser (object) → system libsqlite3 |
MatrixTests.path (json → sqlite) |
| json | duckdb | — | supported | structural | tables | DuckDB read_json (top-level array) / engine JSON parser (object) → DuckDB CREATE TABLE AS |
MatrixTests.path (json → duckdb) |
| json | xml | — | supported | structural | engine JSON codec → XMLParser / engine XML writer | TreeTests.path (json → xml) |
|
| json | yaml | — | supported | lossless | engine JSON codec → Yams | TreeTests.path (json → yaml) |
|
| json | toml | — | supported | structural | engine JSON codec → engine TOML codec | TreeTests.path (json → toml) |
|
| json | plist | — | supported | structural | engine JSON codec → PropertyListSerialization | TreeTests.path (json → plist) |
|
| jsonl | xlsx | — | supported | structural | tables | DuckDB read_json → engine XLSX writer |
MatrixTests.path (jsonl → xlsx) |
| jsonl | xls | — | supported | structural | tables | DuckDB read_json → engine XLSX writer → LibreOffice |
MatrixTests.path (jsonl → xls) |
| jsonl | ods | — | supported | structural | tables | DuckDB read_json → engine ODS writer |
MatrixTests.path (jsonl → ods) |
| jsonl | csv | — | supported | structural | tables | DuckDB read_json → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (jsonl → csv) |
| jsonl | tsv | — | supported | structural | tables | DuckDB read_json → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (jsonl → tsv) |
| jsonl | json | — | supported | structural | tables | DuckDB read_json → DuckDB COPY … (FORMAT json) |
MatrixTests.path (jsonl → json) |
| jsonl | jsonl | — | supported | structural | tables | DuckDB read_json → DuckDB COPY … (FORMAT json) |
MatrixTests.path (jsonl → jsonl) |
| jsonl | parquet | — | supported | structural | tables | DuckDB read_json → DuckDB COPY … (FORMAT parquet) |
MatrixTests.path (jsonl → parquet) |
| jsonl | arrow | — | supported | structural | tables | DuckDB read_json → engine Arrow IPC writer |
MatrixTests.path (jsonl → arrow) |
| jsonl | sqlite | — | supported | structural | tables | DuckDB read_json → system libsqlite3 |
MatrixTests.path (jsonl → sqlite) |
| jsonl | duckdb | — | supported | structural | tables | DuckDB read_json → DuckDB CREATE TABLE AS |
MatrixTests.path (jsonl → duckdb) |
| jsonl | xml | — | supported | structural | tables | DuckDB read_json → engine XML writer |
MatrixTests.path (jsonl → xml) |
| jsonl | yaml | — | supported | structural | tables | DuckDB read_json → Yams (row by row) |
MatrixTests.path (jsonl → yaml) |
| jsonl | toml | — | supported | structural | tables | DuckDB read_json → engine TOML writer |
MatrixTests.path (jsonl → toml) |
| jsonl | plist | — | supported | structural | tables | DuckDB read_json → PropertyListSerialization |
MatrixTests.path (jsonl → plist) |
| parquet | xlsx | — | supported | structural | tables | DuckDB read_parquet → engine XLSX writer |
MatrixTests.path (parquet → xlsx) |
| parquet | xls | — | supported | structural | tables | DuckDB read_parquet → engine XLSX writer → LibreOffice |
MatrixTests.path (parquet → xls) |
| parquet | ods | — | supported | structural | tables | DuckDB read_parquet → engine ODS writer |
MatrixTests.path (parquet → ods) |
| parquet | csv | — | supported | structural | tables | DuckDB read_parquet → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (parquet → csv) |
| parquet | tsv | — | supported | structural | tables | DuckDB read_parquet → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (parquet → tsv) |
| parquet | json | — | supported | structural | tables | DuckDB read_parquet → DuckDB COPY … (FORMAT json) |
MatrixTests.path (parquet → json) |
| parquet | jsonl | — | supported | structural | tables | DuckDB read_parquet → DuckDB COPY … (FORMAT json) |
MatrixTests.path (parquet → jsonl) |
| parquet | parquet | — | supported | lossless | tables | DuckDB read_parquet → DuckDB COPY … (FORMAT parquet) |
MatrixTests.path (parquet → parquet) |
| parquet | arrow | — | supported | structural | tables | DuckDB read_parquet → engine Arrow IPC writer |
MatrixTests.path (parquet → arrow) |
| parquet | sqlite | — | supported | structural | tables | DuckDB read_parquet → system libsqlite3 |
MatrixTests.path (parquet → sqlite) |
| parquet | duckdb | — | supported | lossless | tables | DuckDB read_parquet → DuckDB CREATE TABLE AS |
MatrixTests.path (parquet → duckdb) |
| parquet | xml | — | supported | structural | tables | DuckDB read_parquet → engine XML writer |
MatrixTests.path (parquet → xml) |
| parquet | yaml | — | supported | structural | tables | DuckDB read_parquet → Yams (row by row) |
MatrixTests.path (parquet → yaml) |
| parquet | toml | — | supported | structural | tables | DuckDB read_parquet → engine TOML writer |
MatrixTests.path (parquet → toml) |
| parquet | plist | — | supported | structural | tables | DuckDB read_parquet → PropertyListSerialization |
MatrixTests.path (parquet → plist) |
| arrow | xlsx | — | supported | structural | tables | engine Arrow IPC reader → staging → engine XLSX writer | MatrixTests.path (arrow → xlsx) |
| arrow | xls | — | supported | structural | tables | engine Arrow IPC reader → staging → engine XLSX writer → LibreOffice | MatrixTests.path (arrow → xls) |
| arrow | ods | — | supported | structural | tables | engine Arrow IPC reader → staging → engine ODS writer | MatrixTests.path (arrow → ods) |
| arrow | csv | — | supported | structural | tables | engine Arrow IPC reader → staging → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (arrow → csv) |
| arrow | tsv | — | supported | structural | tables | engine Arrow IPC reader → staging → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (arrow → tsv) |
| arrow | json | — | supported | structural | tables | engine Arrow IPC reader → staging → DuckDB COPY … (FORMAT json) |
MatrixTests.path (arrow → json) |
| arrow | jsonl | — | supported | structural | tables | engine Arrow IPC reader → staging → DuckDB COPY … (FORMAT json) |
MatrixTests.path (arrow → jsonl) |
| arrow | parquet | — | supported | structural | tables | engine Arrow IPC reader → staging → DuckDB COPY … (FORMAT parquet) |
MatrixTests.path (arrow → parquet) |
| arrow | sqlite | — | supported | structural | tables | engine Arrow IPC reader → staging → system libsqlite3 | MatrixTests.path (arrow → sqlite) |
| arrow | duckdb | — | supported | structural | tables | engine Arrow IPC reader → staging → DuckDB CREATE TABLE AS |
MatrixTests.path (arrow → duckdb) |
| arrow | xml | — | supported | structural | tables | engine Arrow IPC reader → staging → engine XML writer | MatrixTests.path (arrow → xml) |
| arrow | yaml | — | supported | structural | tables | engine Arrow IPC reader → staging → Yams (row by row) | MatrixTests.path (arrow → yaml) |
| arrow | toml | — | supported | structural | tables | engine Arrow IPC reader → staging → engine TOML writer | MatrixTests.path (arrow → toml) |
| arrow | plist | — | supported | structural | tables | engine Arrow IPC reader → staging → PropertyListSerialization | MatrixTests.path (arrow → plist) |
| sqlite | xlsx | — | supported | structural | multipleSheets, tables | system libsqlite3 → staging → engine XLSX writer | MatrixTests.path (sqlite → xlsx) |
| sqlite | xls | — | supported | structural | multipleSheets, tables | system libsqlite3 → staging → engine XLSX writer → LibreOffice | MatrixTests.path (sqlite → xls) |
| sqlite | ods | — | supported | structural | multipleSheets, tables | system libsqlite3 → staging → engine ODS writer | MatrixTests.path (sqlite → ods) |
| sqlite | csv | — | supported | partial | tables | system libsqlite3 → staging → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (sqlite → csv) |
| sqlite | tsv | — | supported | partial | tables | system libsqlite3 → staging → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (sqlite → tsv) |
| sqlite | json | — | supported | partial | tables | system libsqlite3 → staging → DuckDB COPY … (FORMAT json) |
MatrixTests.path (sqlite → json) |
| sqlite | jsonl | — | supported | partial | tables | system libsqlite3 → staging → DuckDB COPY … (FORMAT json) |
MatrixTests.path (sqlite → jsonl) |
| sqlite | parquet | — | supported | partial | tables | system libsqlite3 → staging → DuckDB COPY … (FORMAT parquet) |
MatrixTests.path (sqlite → parquet) |
| sqlite | arrow | — | supported | partial | tables | system libsqlite3 → staging → engine Arrow IPC writer | MatrixTests.path (sqlite → arrow) |
| sqlite | duckdb | — | supported | structural | multipleSheets, tables | system libsqlite3 → staging → DuckDB CREATE TABLE AS |
MatrixTests.path (sqlite → duckdb) |
| sqlite | xml | — | supported | partial | tables | system libsqlite3 → staging → engine XML writer | MatrixTests.path (sqlite → xml) |
| sqlite | yaml | — | supported | partial | tables | system libsqlite3 → staging → Yams (row by row) | MatrixTests.path (sqlite → yaml) |
| sqlite | toml | — | supported | partial | tables | system libsqlite3 → staging → engine TOML writer | MatrixTests.path (sqlite → toml) |
| sqlite | plist | — | supported | partial | tables | system libsqlite3 → staging → PropertyListSerialization | MatrixTests.path (sqlite → plist) |
| duckdb | xlsx | — | supported | structural | multipleSheets, tables | DuckDB ATTACH (read-only) → engine XLSX writer |
MatrixTests.path (duckdb → xlsx) |
| duckdb | xls | — | supported | structural | multipleSheets, tables | DuckDB ATTACH (read-only) → engine XLSX writer → LibreOffice |
MatrixTests.path (duckdb → xls) |
| duckdb | ods | — | supported | structural | multipleSheets, tables | DuckDB ATTACH (read-only) → engine ODS writer |
MatrixTests.path (duckdb → ods) |
| duckdb | csv | — | supported | partial | tables | DuckDB ATTACH (read-only) → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (duckdb → csv) |
| duckdb | tsv | — | supported | partial | tables | DuckDB ATTACH (read-only) → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (duckdb → tsv) |
| duckdb | json | — | supported | partial | tables | DuckDB ATTACH (read-only) → DuckDB COPY … (FORMAT json) |
MatrixTests.path (duckdb → json) |
| duckdb | jsonl | — | supported | partial | tables | DuckDB ATTACH (read-only) → DuckDB COPY … (FORMAT json) |
MatrixTests.path (duckdb → jsonl) |
| duckdb | parquet | — | supported | partial | tables | DuckDB ATTACH (read-only) → DuckDB COPY … (FORMAT parquet) |
MatrixTests.path (duckdb → parquet) |
| duckdb | arrow | — | supported | partial | tables | DuckDB ATTACH (read-only) → engine Arrow IPC writer |
MatrixTests.path (duckdb → arrow) |
| duckdb | sqlite | — | supported | structural | multipleSheets, tables | DuckDB ATTACH (read-only) → system libsqlite3 |
MatrixTests.path (duckdb → sqlite) |
| duckdb | xml | — | supported | partial | tables | DuckDB ATTACH (read-only) → engine XML writer |
MatrixTests.path (duckdb → xml) |
| duckdb | yaml | — | supported | partial | tables | DuckDB ATTACH (read-only) → Yams (row by row) |
MatrixTests.path (duckdb → yaml) |
| duckdb | toml | — | supported | partial | tables | DuckDB ATTACH (read-only) → engine TOML writer |
MatrixTests.path (duckdb → toml) |
| duckdb | plist | — | supported | partial | tables | DuckDB ATTACH (read-only) → PropertyListSerialization |
MatrixTests.path (duckdb → plist) |
| xml | xlsx | — | supported | structural | tables | XMLParser → records → engine XLSX writer | MatrixTests.path (xml → xlsx) |
| xml | xls | — | supported | structural | tables | XMLParser → records → engine XLSX writer → LibreOffice | MatrixTests.path (xml → xls) |
| xml | ods | — | supported | structural | tables | XMLParser → records → engine ODS writer | MatrixTests.path (xml → ods) |
| xml | csv | — | supported | structural | tables | XMLParser → records → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (xml → csv) |
| xml | tsv | — | supported | structural | tables | XMLParser → records → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (xml → tsv) |
| xml | json | — | supported | structural | XMLParser / engine XML writer → engine JSON codec | TreeTests.path (xml → json) |
|
| xml | jsonl | — | supported | structural | tables | XMLParser → records → DuckDB COPY … (FORMAT json) |
MatrixTests.path (xml → jsonl) |
| xml | parquet | — | supported | structural | tables | XMLParser → records → DuckDB COPY … (FORMAT parquet) |
MatrixTests.path (xml → parquet) |
| xml | arrow | — | supported | structural | tables | XMLParser → records → engine Arrow IPC writer | MatrixTests.path (xml → arrow) |
| xml | sqlite | — | supported | structural | tables | XMLParser → records → system libsqlite3 | MatrixTests.path (xml → sqlite) |
| xml | duckdb | — | supported | structural | tables | XMLParser → records → DuckDB CREATE TABLE AS |
MatrixTests.path (xml → duckdb) |
| xml | xml | — | supported | structural | XMLParser / engine XML writer → XMLParser / engine XML writer | TreeTests.path (xml → xml) |
|
| xml | yaml | — | supported | structural | XMLParser / engine XML writer → Yams | TreeTests.path (xml → yaml) |
|
| xml | toml | — | supported | structural | XMLParser / engine XML writer → engine TOML codec | TreeTests.path (xml → toml) |
|
| xml | plist | — | supported | structural | XMLParser / engine XML writer → PropertyListSerialization | TreeTests.path (xml → plist) |
|
| yaml | xlsx | — | supported | structural | tables | Yams → records → engine XLSX writer | MatrixTests.path (yaml → xlsx) |
| yaml | xls | — | supported | structural | tables | Yams → records → engine XLSX writer → LibreOffice | MatrixTests.path (yaml → xls) |
| yaml | ods | — | supported | structural | tables | Yams → records → engine ODS writer | MatrixTests.path (yaml → ods) |
| yaml | csv | — | supported | structural | tables | Yams → records → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (yaml → csv) |
| yaml | tsv | — | supported | structural | tables | Yams → records → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (yaml → tsv) |
| yaml | json | — | supported | structural | Yams → engine JSON codec | TreeTests.path (yaml → json) |
|
| yaml | jsonl | — | supported | structural | tables | Yams → records → DuckDB COPY … (FORMAT json) |
MatrixTests.path (yaml → jsonl) |
| yaml | parquet | — | supported | structural | tables | Yams → records → DuckDB COPY … (FORMAT parquet) |
MatrixTests.path (yaml → parquet) |
| yaml | arrow | — | supported | structural | tables | Yams → records → engine Arrow IPC writer | MatrixTests.path (yaml → arrow) |
| yaml | sqlite | — | supported | structural | tables | Yams → records → system libsqlite3 | MatrixTests.path (yaml → sqlite) |
| yaml | duckdb | — | supported | structural | tables | Yams → records → DuckDB CREATE TABLE AS |
MatrixTests.path (yaml → duckdb) |
| yaml | xml | — | supported | structural | Yams → XMLParser / engine XML writer | TreeTests.path (yaml → xml) |
|
| yaml | yaml | — | supported | structural | Yams → Yams | TreeTests.path (yaml → yaml) |
|
| yaml | toml | — | supported | structural | Yams → engine TOML codec | TreeTests.path (yaml → toml) |
|
| yaml | plist | — | supported | structural | Yams → PropertyListSerialization | TreeTests.path (yaml → plist) |
|
| toml | xlsx | — | supported | structural | tables | engine TOML parser → records → engine XLSX writer | MatrixTests.path (toml → xlsx) |
| toml | xls | — | supported | structural | tables | engine TOML parser → records → engine XLSX writer → LibreOffice | MatrixTests.path (toml → xls) |
| toml | ods | — | supported | structural | tables | engine TOML parser → records → engine ODS writer | MatrixTests.path (toml → ods) |
| toml | csv | — | supported | structural | tables | engine TOML parser → records → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (toml → csv) |
| toml | tsv | — | supported | structural | tables | engine TOML parser → records → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (toml → tsv) |
| toml | json | — | supported | structural | engine TOML codec → engine JSON codec | TreeTests.path (toml → json) |
|
| toml | jsonl | — | supported | structural | tables | engine TOML parser → records → DuckDB COPY … (FORMAT json) |
MatrixTests.path (toml → jsonl) |
| toml | parquet | — | supported | structural | tables | engine TOML parser → records → DuckDB COPY … (FORMAT parquet) |
MatrixTests.path (toml → parquet) |
| toml | arrow | — | supported | structural | tables | engine TOML parser → records → engine Arrow IPC writer | MatrixTests.path (toml → arrow) |
| toml | sqlite | — | supported | structural | tables | engine TOML parser → records → system libsqlite3 | MatrixTests.path (toml → sqlite) |
| toml | duckdb | — | supported | structural | tables | engine TOML parser → records → DuckDB CREATE TABLE AS |
MatrixTests.path (toml → duckdb) |
| toml | xml | — | supported | structural | engine TOML codec → XMLParser / engine XML writer | TreeTests.path (toml → xml) |
|
| toml | yaml | — | supported | structural | engine TOML codec → Yams | TreeTests.path (toml → yaml) |
|
| toml | toml | — | supported | structural | engine TOML codec → engine TOML codec | TreeTests.path (toml → toml) |
|
| toml | plist | — | supported | structural | engine TOML codec → PropertyListSerialization | TreeTests.path (toml → plist) |
|
| plist | xlsx | — | supported | structural | tables | PropertyListSerialization → records → engine XLSX writer | MatrixTests.path (plist → xlsx) |
| plist | xls | — | supported | structural | tables | PropertyListSerialization → records → engine XLSX writer → LibreOffice | MatrixTests.path (plist → xls) |
| plist | ods | — | supported | structural | tables | PropertyListSerialization → records → engine ODS writer | MatrixTests.path (plist → ods) |
| plist | csv | — | supported | structural | tables | PropertyListSerialization → records → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (plist → csv) |
| plist | tsv | — | supported | structural | tables | PropertyListSerialization → records → DuckDB COPY … (FORMAT csv) |
MatrixTests.path (plist → tsv) |
| plist | json | — | supported | structural | PropertyListSerialization → engine JSON codec | TreeTests.path (plist → json) |
|
| plist | jsonl | — | supported | structural | tables | PropertyListSerialization → records → DuckDB COPY … (FORMAT json) |
MatrixTests.path (plist → jsonl) |
| plist | parquet | — | supported | structural | tables | PropertyListSerialization → records → DuckDB COPY … (FORMAT parquet) |
MatrixTests.path (plist → parquet) |
| plist | arrow | — | supported | structural | tables | PropertyListSerialization → records → engine Arrow IPC writer | MatrixTests.path (plist → arrow) |
| plist | sqlite | — | supported | structural | tables | PropertyListSerialization → records → system libsqlite3 | MatrixTests.path (plist → sqlite) |
| plist | duckdb | — | supported | structural | tables | PropertyListSerialization → records → DuckDB CREATE TABLE AS |
MatrixTests.path (plist → duckdb) |
| plist | xml | — | supported | structural | PropertyListSerialization → XMLParser / engine XML writer | TreeTests.path (plist → xml) |
|
| plist | yaml | — | supported | structural | PropertyListSerialization → Yams | TreeTests.path (plist → yaml) |
|
| plist | toml | — | supported | structural | PropertyListSerialization → engine TOML codec | TreeTests.path (plist → toml) |
|
| plist | plist | — | supported | lossless | PropertyListSerialization → PropertyListSerialization | TreeTests.path (plist → plist) |
Limitations shown to the user (verbatim from the engine, with the paths they apply to)
- Only cell values are converted. Formulas (their last calculated values are kept), styles, number formats, column widths, charts, images, pivot tables, comments, data validation and conditional formatting are not carried over. — from ods, xls, xlsx (42 paths)
- Merged cells keep their value in the first cell only. — from ods, xls, xlsx (42 paths)
- Spreadsheet numbers are 64-bit floats: integers beyond 2^53 and long decimals are written as text; NaN and infinities become empty cells; time-zoned instants are written as UTC wall-clock times. — to ods, xls, xlsx (42 paths)
- Date-times are stored to the millisecond. — to xls, xlsx (28 paths)
- At most 65,536 rows and 256 columns per sheet (legacy format limit); produced through the bundled LibreOffice. — to xls (14 paths)
- At most 1,048,576 rows and 16,384 columns per sheet (larger tables stop with an error, or are split across sheets on request). — to ods, xlsx (28 paths)
- One sheet per output file: the first one by default, a chosen one, or all of them as separate files. — from ods, xls, xlsx (30 paths)
- Text formats carry no types: dates are written in ISO 8601, binary values as Base64, nested values as JSON text. — to csv, tsv (30 paths)
- JSON has no date, binary, NaN or infinity: dates become ISO 8601 text, binary values Base64 text, NaN and infinities null. — to json, jsonl (25 paths)
- Columns are written as int64, float64, bool, utf8, binary, date32, time64[us], timestamp[us] and decimal128; nested values as JSON text. Buffers are not compressed. — to arrow (14 paths)
- SQLite numbers are 64-bit: exact decimals and 128-bit integers are stored as TEXT; dates and times as ISO 8601 text in DATE / DATETIME / TIME columns. — to sqlite (14 paths)
- XML carries text only; empty values are omitted, so empty text and missing values cannot be told apart when read back. — to xml (10 paths)
- Binary values are written as Base64 text. — to yaml (10 paths)
- TOML has no null: empty values are omitted (or stop the conversion, as chosen). — to toml (10 paths)
- Property lists have no null (empty values are omitted or stop the conversion) and are built in memory (20 million cells at most). — to plist (10 paths)
- Column types are detected from the text (verified on every row); numbers are re-written in canonical form (1.50 → 1.5). Identifiers with leading zeros stay text. Turn detection off to keep every cell as written. — from csv, tsv, xml (40 paths)
- Same-format conversion: the file is re-written with the chosen options. — csv→csv, tsv→tsv, jsonl→jsonl, parquet→parquet
- Integers beyond 64 bits are read as 64-bit floats by the JSON reader. Text that looks like an ISO date becomes a date. — from json, jsonl (25 paths)
- The table is built from the document's list of records (detected, or given by “Record list”); other content of the document is not converted. — from json, plist, toml, xml, yaml (50 paths)
- XML carries text only: numbers and booleans become text, empty strings and nulls both become empty elements. Keys starting with “@” become attributes, “#text” becomes element text, lists become repeated elements. — json→xml, xml→xml, yaml→xml, toml→xml, plist→xml
- TOML has no null (nulls are omitted or stop the conversion, as chosen) and needs a table at the top level; plain values are moved before sub-tables. — json→toml, xml→toml, yaml→toml, toml→toml, plist→toml
- Property lists have no null and no key order (keys are sorted when read back); integers beyond 64 bits and exact decimals are written as text. — json→plist, xml→plist, yaml→plist, toml→plist, plist→plist
- ZSTD-compressed Arrow buffers, union, run-end-encoded and view types are not supported. Nested columns are read as JSON text; unsigned 64-bit integers become exact decimals. — from arrow (14 paths)
- Only tables are read. Views, indexes, constraints and triggers are not carried over; stored SQL is never executed. — from duckdb, sqlite (28 paths)
- One table per output file: the first one by default, a chosen one, or all of them as separate files. — from duckdb, sqlite (20 paths)
- Dates and binary values become text (ISO 8601 / Base64); JSON has no such types. — xml→json, yaml→json, toml→json, plist→json
- XML comments and processing instructions are dropped; attribute order is not kept; every value is read as text. Documents declaring entities (DTD) are refused. — xml→json, xml→xml, xml→yaml, xml→toml, xml→plist
- The document is re-serialised: comments and original formatting are not kept. — xml→xml, yaml→yaml, toml→toml
- Comments, anchors and formatting of the source are not kept. — xml→yaml, yaml→yaml, toml→yaml, plist→yaml
- YAML comments, anchors and tags are not carried over. — yaml→json, yaml→xml, yaml→toml, yaml→plist
- TOML comments are not carried over. — toml→json, toml→xml, toml→yaml, toml→toml, toml→plist
3. Type mapping
Internal column kinds: string, integer (64-bit), decimal (exact digits), double, boolean, date, timestamp (no zone), timestampTZ (UTC instant), time, binary, json (nested value). Dates/times are integers (days, microseconds):
no calendar or time-zone arithmetic is applied, so nothing is shifted.
Reading
| Source | integer | decimal | double | boolean | date / timestamp | text | other |
|---|---|---|---|---|---|---|---|
| CSV / TSV / XML records | text -?(0|[1-9][0-9]*) in 64-bit range → BIGINT; wider → DECIMAL(38,0) |
> 15 significant digits → DECIMAL(38,s) | other numbers (optional decimal comma / thousands separator) | true/false (any case) |
ISO 8601 date, date-time (→ TIMESTAMP), date-time with offset (→ TIMESTAMPTZ, instant kept), HH:MM:SS → TIME; d/m/y or m/d/y only when unambiguous or when a format is given |
everything else, incl. leading-zero identifiers, +33, mixed columns |
detection is verified on every row; one non-conforming value keeps the column text |
| JSON / JSON Lines | BIGINT (UBIGINT/HUGEINT when needed) | — | DOUBLE | BOOLEAN | ISO date/time strings → DATE/TIMESTAMP/TIMESTAMPTZ/TIME (verified on every row) | VARCHAR | objects → STRUCT, arrays → LIST (kept, flattened or JSON text, see §5); integers beyond 64 bits → DOUBLE (DuckDB reader) |
| YAML / TOML / plist / JSON object | as JSON; integers beyond 64 bits and long decimals are carried as text (exact) | native dates of the format | plist Data, YAML !!binary → Base64 text |
||||
| Parquet / DuckDB | native DuckDB types, unchanged | nested types kept / flattened / JSON text | |||||
| XLSX / XLS | number cell whose text is plain digits | — | other number cells | t="b" |
number + date/time number format (built-in ids and custom codes), 1900 and 1904 systems; t="d" ISO cells |
shared, inline, formula-string cells | errors → NULL (warned); mixed column → text |
| ODS | float/percentage/currency with integer text |
— | other floats | boolean |
date (date or date-time), time (duration < 24 h) |
string |
mixed column → text |
| SQLite | INTEGER storage | — | REAL | INTEGER 0/1 in a column declared BOOL* |
TEXT ISO values in columns declared DATE*/TIME* |
TEXT | BLOB → binary; a column mixing storage classes → text |
| Arrow | int8…int64, uint8…uint32 → BIGINT; uint64 → UBIGINT | decimal128 → DECIMAL(p,s) | float16/32/64 | bool | date32/64, time32/64, timestamp (s/ms/µs/ns; ns truncated to µs with a warning; zoned → TIMESTAMPTZ) | utf8, large_utf8, dictionary-encoded strings | binary, large/fixed-size binary; list/struct/map/fixed-size-list → JSON text; duration → BIGINT |
Writing
| Target | integer | decimal / 128-bit | double | boolean | date | timestamp | timestampTZ | time | binary | nested | NULL vs "" |
|---|---|---|---|---|---|---|---|---|---|---|---|
| CSV / TSV | digits | exact digits | shortest round-trip | true/false |
ISO (or strftime option) |
YYYY-MM-DDTHH:MM:SS[.ffffff] |
same + Z |
ISO | Base64 | JSON text | NULL = empty field (or chosen text), "" = "" |
| JSON / JSON Lines | number | number, digit for digit | number; NaN/±inf → null (counted warning) |
true/false |
ISO string | ISO string | ISO string + Z |
string | Base64 string | real JSON | null vs "" |
| Parquet | INT64 etc. (native) | DECIMAL; 128-bit ints → DECIMAL(38,0), or text if 39 digits | DOUBLE/FLOAT | BOOLEAN | DATE | TIMESTAMP (µs) | TIMESTAMP (UTC) | TIME | BYTE_ARRAY | native STRUCT/LIST/MAP | native |
| DuckDB | native | native | native | native | native | native | native | native | BLOB | native | native |
| Arrow | int64 | decimal128(p,s); 128-bit ints as Parquet | float64 | bool | date32 | timestamp[us] | timestamp[us, UTC] | time64[us] | binary | utf8 (JSON text) | validity bitmap |
| SQLite | INTEGER | TEXT (exact) | REAL (NaN → text nan) |
BOOLEAN 0/1 |
DATE ISO text |
DATETIME ISO text |
DATETIME ISO text + Z |
TIME text |
BLOB | TEXT (JSON) | NULL vs '' |
| XLSX / XLS | number; beyond ±2^53 → text (option: number) | ≤ 15 digits → number, else text | number; NaN/±inf → empty (counted) | boolean | date cell (yyyy-mm-dd), outside 1900-03-01…9999 → ISO text |
date-time cell, millisecond precision (finer → rounded, counted) | UTC wall clock, zone dropped (warned) | time cell | Base64 text | JSON text | empty cell vs empty string cell (XLS: both empty) |
| ODS | float |
as XLSX | float |
boolean |
date |
date with full microseconds |
UTC wall clock (warned) | time duration |
Base64 text | JSON text | empty cell vs empty string cell |
| XML (records) | text | text | text | text | text | text | text | text | Base64 | JSON text → text | NULL = element omitted; "" reads back as NULL |
| YAML (list of mappings) | int | float literal, exact digits | float (.nan, .inf) |
bool | timestamp | timestamp | timestamp Z |
quoted string | Base64 string | real nested YAML | null vs "" |
TOML ([[rows]]) |
integer | float literal / string for > 64-bit integers | float (nan, inf) |
boolean | local date | local date-time | offset date-time | local time | Base64 string | inline tables / arrays | NULL omitted (or error) |
| plist (array of dicts) | integer | string (exact) | real | boolean | string | string | Date (XML plists: to the second, warned) |
string | Data |
nested dict/array | NULL omitted (or error) |
4. Reading and writing options (all exposed as OptionDescriptors)
Reading: encoding.input, csv.delimiter.input, csv.quote.input, csv.lenient, nulls.input, header
(auto/yes/no), header.row, types.detect (off = all text), types.columns (name=text|integer|decimal|boolean| date|timestamp), numbers.decimal.comma, numbers.thousands, dates.input.format, sheet (name or #n),
sheets.all, formulas.as.text, records.path, nested (auto/flatten/json/native), columns.select,
columns.rename.
Writing: csv.preset (standard / Excel-friendly = UTF-8 BOM + CRLF), csv.delimiter, csv.header,
csv.quote.all, csv.line.ending, encoding.output, encoding.bom, nulls.output, dates.output.format,
formula.guard (CSV-injection guard, off by default: prefixes text cells starting with = + - @ with ',
counted and reported), json.pretty, xml.root.name, xml.row.name, xml.item.name, nulls.policy
(omit/error, TOML and plist), records.key, plist.format, parquet.compression
(zstd/snappy/gzip/lz4/uncompressed), parquet.row.group.size, arrow.format (file/stream),
sheet.header.bold, sheet.header.freeze, sheet.overflow (error/split), integers.big, xlsx.strings,
table.name, memory.limit.mb, threads.
- Encoding detection: BOM (UTF-8, UTF-16 LE/BE) → UTF-8 validity → UTF-16 without BOM (NUL pattern) →
single-byte heuristic (no 0x80–0x9F bytes: ISO-8859-1; bytes undefined in Windows-1252 or accented letters inside
words: Mac Roman; otherwise Windows-1252). A heuristic answer raises the
encoding-guessedwarning. Non-UTF-8 input is transcoded to UTF-8 in the workspace before DuckDB reads it. - Delimiter: engine sniffer over comma, semicolon, tab, pipe (quote-aware, first 64 KB); quote/escape/header:
DuckDB's sniffer for that delimiter. Rows of unequal length stop the conversion with an explanation unless
csv.lenientis on. - NULL on input: an unquoted empty field is NULL,
""is an empty string; extra NULL tokens vianulls.inputapply to unquoted fields only. - Sheets/tables: to a workbook or database target, every sheet/table is carried by default; to a single-table
target, the first visible one (others listed by name in a
sheets-droppeddata-loss warning) or all of them as<name>-<sheet>.<ext>files.
5. Nested values (JSON, Parquet, DuckDB, Arrow, YAML/TOML/plist/XML records)
nested |
effect |
|---|---|
flatten (default for flat targets) |
each STRUCT field becomes a column parent.child (recursively); lists and maps become JSON text |
json |
every nested column becomes one JSON-text column |
native (default for Parquet, DuckDB, JSON, JSON Lines targets) |
nested types are kept as they are |
Arrow nested columns are always read as JSON text. A JSON-text column written to JSON/YAML/TOML/plist is re-expanded into real nested values.
6. No silent loss — warnings
preflight opens the real file exactly as convert would (first 20 000 rows per table; counts taken from a partial
read say so) and adds the target's schema-level effects; convert raises the same warnings with exact counts.
Codes: sheets-dropped (names), formulas-dropped / formulas-as-text (count), error-cells, merged-cells,
hidden-rows-columns, hidden-sheet, sheet-features-dropped (data validation, conditional formatting,
hyperlinks), workbook-content-dropped (charts, images, pivot tables, comments, external links, macros, defined
names — detected from the package parts), blank-rows-skipped, mixed-types, columns-renamed,
columns-not-selected, encoding-guessed, undecodable-characters, unencodable-characters,
ambiguous-date-format, type-detection-reverted, timezone-normalized, timezone-dropped,
sub-millisecond-dropped, sub-microsecond-dropped, plist-subsecond-dropped, integer-as-text,
decimal-as-text, int128-as-decimal, int128-as-text, wide-number-as-text, non-finite-dropped,
date-as-text, time-as-text, binary-as-text, cell-too-long, csv-injection-guarded, nested-flattened,
nested-stringified, json-number-precision, spreadsheet-number-precision, spreadsheet-time-precision,
content-outside-records, records-located, single-record, duplicate-keys, non-string-keys,
yaml-multiple-documents, xml-comments-dropped, xml-processing-instructions-dropped, xml-mixed-content,
xml-sibling-order, xml-names-adapted, xml-empty-strings, xml-types-as-text, toml-root-wrapped,
key-order-changed, null-omitted, control-characters-replaced, views-not-converted, sheets-renamed,
tables-renamed, sheet-split, xls-via-libreoffice, xls-column-limit.
validate re-opens the output in a fresh locked-down session and checks: it opens; number of tables; exact row
count read back vs rows written; column count; column names; and the cells of the first 1 000 rows of each table
against an independent read of the source (numbers compared by value, date-times by instant). On lossless paths a
single differing cell fails validation; on the others the comparison is reported (test validationCatchesAWrongOutput).
Tree paths: the output is parsed back; lossless paths must be identical, others report the comparison of all
scalar values.
inspect returns: sheets/tables with rows × columns (workbooks are scanned up to 96 MB of compressed sheet data;
beyond that counts are "at least"), formulas, merged ranges, hidden rows/columns, charts/images/pivots/macros;
text encoding, delimiter, quote, header, detected column types; Parquet created_by, row groups, format version,
compression codecs and physical/logical schema; SQLite page size, page count, user_version, application_id,
encoding, journal mode, tables/views with row counts and declared column types; DuckDB table list with row counts
and schema (and the DuckDB version used to read it); Arrow variant, row count and Arrow types; for documents, the
top-level shape, the detected record list and the reader's notes.
7. Security limits
| Limit | Value | Behaviour |
|---|---|---|
| ZIP entries | 200 000 | limitExceeded |
| Small package part (workbook, styles, rels) | 256 MB decompressed | limitExceeded |
| Streamed part (worksheet, content.xml, shared strings) | 16 GB decompressed, counted on bytes actually produced, never on declared sizes; CRC-32 verified | limitExceeded / malformedInput |
| Shared strings in memory | 1 GB | limitExceeded |
| Sheet | 1 048 576 rows × 16 384 columns | limitExceeded |
| ODS repeated rows/cells | materialised cells ≤ 500 million; trailing empty repeats cost nothing; a repeat that cannot fit is refused before any row is produced | limitExceeded |
| XML token (package parts) | 128 MB | limitExceeded |
| DOCTYPE / entities | refused everywhere: package parts (tokenizer has no DTD support) and XML documents (byte pre-check + parser delegate; shouldResolveExternalEntities = false) |
malformedInput |
| Tree documents | 512 MB file, 256 levels, 50 million values; YAML alias expansion bounded by input size | limitExceeded |
| SQLite | read-only, defensive mode, trusted_schema off, views/triggers disabled, no ATTACH, value ≤ 512 MB; only tables are read |
error |
| Arrow | message header ≤ 256 MB, decompressed buffer ≤ 2 GB, every offset/length checked | error |
| Encrypted input | OLE-wrapped (password-protected) XLSX, encrypted ZIP entries, encrypted ODS | encryptedInput |
| Macros | never executed (VBA parts are only counted) | — |
8. Performance (measured)
Release build (swift test -c release -Xswiftc -enable-testing --filter Benchmarks, harness
Tests/EngineDataTests/Benchmarks.swift), one process per benchmark so that the peak resident size (ru_maxrss) is
that benchmark's. Inputs generated by DuckDB: 9 columns (integer id, 32-char token, text, decimal amount, boolean,
date, timestamp, category, zero-padded zip). Times include type detection + its full-file verification pass and
exclude validate. Defaults: memory limit 4 096 MB, 8 threads, unless stated.
The Mac was shared with other heavy builds and test runs during these measurements (1-minute load average between 23 and 84 on 18 cores). The numbers are what was observed, not best cases; they vary from run to run.
| Conversion | Rows | Source | Output | Time | Rows/s | Source MB/s | Peak RSS | validate |
Load avg (1 min) |
|---|---|---|---|---|---|---|---|---|---|
| CSV (2.36 GB) → Parquet, zstd | 22 000 000 | 2 363 MB | 538 MB | 6.90 s | 3 187 274 | 342 | 1 558 MB | 0.29 s, all passed | 83.84 |
| CSV (2.36 GB) → Parquet, snappy | 22 000 000 | 2 363 MB | 1 226 MB | 4.57 s | 4 818 602 | 518 | 1 403 MB | 0.30 s, all passed | 45.57 |
| CSV (2.36 GB) → Parquet, zstd, memory limit 1 024 MB | 22 000 000 | 2 363 MB | 538 MB | 8.25 s | 2 666 335 | 286 | 1 210 MB | 0.24 s, all passed | 44.72 |
| CSV (2.36 GB) → Parquet, zstd, memory limit 512 MB, 4 threads | 22 000 000 | 2 363 MB | 538 MB | 7.49 s | 2 936 470 | 315 | 812 MB | 0.16 s, all passed | 44.74 |
| CSV (2.36 GB) → Parquet, zstd, type detection off | 22 000 000 | 2 363 MB | 462 MB | 2.43 s | 9 067 708 | 974 | 1 961 MB | 0.17 s, all passed | 40.77 |
| Parquet (zstd) → CSV | 22 000 000 | 538 MB | 2 363 MB | 3.10 s | 7 096 124 | 174 | 2 071 MB | 1.13 s, all passed | 40.77 |
| Parquet (zstd) → CSV, memory limit 512 MB, 4 threads | 22 000 000 | 538 MB | 2 363 MB | 4.17 s | 5 276 365 | 129 | 758 MB | 1.35 s, all passed | 46.02 |
| CSV → XLSX (1 M rows, shared strings) | 1 000 000 | 105 MB | 73 MB | 6.16 s | 162 428 | 17 | 384 MB | 4.66 s, all passed | 45.38 |
| CSV → XLSX (1 M rows, inline strings) | 1 000 000 | 105 MB | 79 MB | 4.44 s | 225 321 | 24 | 207 MB | 3.87 s, all passed | 45.43 |
| XLSX (1 M rows) → CSV | 1 000 000 | 73 MB | 105 MB | 4.55 s | 219 933 | 16 | 552 MB | 1.00 s, all passed | 39.44 |
| CSV → ODS (1 M rows) | 1 000 000 | 105 MB | 73 MB | 5.43 s | 184 246 | 20 | 206 MB | 8.83 s, all passed | 37.65 |
| JSON Lines (1.45 GB) → Parquet, zstd | 8 000 000 | 1 480 MB | 195 MB | 12.99 s | 615 644 | 114 | 3 685 MB | 0.46 s, all passed | 31.53 |
| CSV (2.36 GB) → SQLite | 22 000 000 | 2 363 MB | 2 431 MB | 33.07 s | 665 267 | 71 | 503 MB | 0.66 s, all passed | 26.86 |
| CSV (2.36 GB) → Arrow IPC file | 22 000 000 | 2 363 MB | 2 094 MB | 13.43 s | 1 637 648 | 176 | 443 MB | 0.20 s, all passed | 22.64 |
Reading the table: CSV → Parquet reads the 2.36 GB file twice (verification of detected types, then the copy); with detection off it is read once. With a 512 MB limit the same 2.36 GB conversions complete in well under the file's size in resident memory, which is the larger-than-memory proof asked for. JSON Lines → Parquet detects the schema on the whole file before converting; its peak of 3.7 GB was measured with the default 4 GB limit and was not measured with a lower one. CSV → SQLite is bound by single-threaded row inserts. XLSX → CSV keeps the workbook's shared strings in memory (1 M distinct 32-character tokens here).
9. Evaluated and NOT implemented
- Apache
arrow-swift(evaluated at commit15dfba80, 2026-11-10): it builds with Swift 6.4.2 (tools 5.10, Swift 5 language mode) but pulls 21 transitive packages (grpc-swift, swift-nio, swift-protobuf… for Arrow Flight), and its reader failed on files written by Arrow C++ (pyarrow 24):Unsupported field type found: largeutf8, and…: decimalon a file with a decimal128 column. Rejected; the engine has its own reader/writer, cross-checked during development against pyarrow 24 in both directions (pyarrowvalidate(full=True)on our files); the tests use four small files written by pyarrow (Tests/EngineDataTests/Fixtures) and a byte-level layout test, and do not need Python. - Arrow: ZSTD-compressed buffers (no public Zstandard decoder on macOS; clear error telling to re-save without ZSTD), union, run-end-encoded, string-view/binary-view, list-view, interval, decimal256, big-endian files, dictionary-encoded or nested columns on write, buffer compression on write, Feather v1.
- DuckDB extensions (sqlite, excel, arrow, spatial, httpfs…): not used by design (no download, no network).
- Native BIFF8 (XLS) parser: not written; XLS goes through the bundled LibreOffice (a
sofficeprocess is started for each conversion; its cost was not benchmarked). Without LibreOffice in the build, XLS paths are not declared. - XLSX writer: formulas, styles beyond the bold frozen header and date formats, column widths, charts, multiple
header rows, ZIP64 (a package part over 4 GB stops with
limitExceeded). XLSX reader: formulas are not evaluated (cached values are read); rich-text formatting, comments and hyperlinks are dropped (reported); XLSB and password-protected workbooks are not read. - ODS: freeze panes on write; formula evaluation; sub-tables.
- Views in SQLite and DuckDB files: listed, never executed. Indexes, constraints and triggers are not carried between databases. SQLite → SQLite and DuckDB → DuckDB are not offered (copy the file).
- XML: DTDs/entities are refused (also harmless ones); namespaces are kept as written prefixes, not resolved; mixed content keeps its text but not its position; documents over 512 MB (tree in memory).
- JSON tabular input: integers beyond 64 bits become doubles (DuckDB reader) for top-level arrays and JSON Lines; exact handling only for documents read by the engine's parser (JSON objects, YAML, TOML, plist, where such numbers are carried as text).
- YAML: comments, anchors, custom tags and block styles are not preserved;
!!binaryis written as a Base64 string (Yams' emitter does not write tags). TOML: comments not preserved. plist: key order (Apple's API). - Text encodings other than UTF-8, UTF-16 LE/BE, Windows-1252, ISO-8859-1, Mac Roman.
- Geospatial and scientific formats (GeoJSON, GPKG, HDF5, NetCDF…): out of scope of this engine by assignment.
9b. XML mapping of a document's top level (fixed 2026-11-21)
An XML document has exactly one root element. JSON, YAML and property lists may hold anything at their top level, so
the tree → XML writer maps the top level as follows (Tree/XMLTree.swift, XMLTreeCodec.Writer.serialize):
| Top-level value | Written as | Reads back (XML → JSON) as |
|---|---|---|
| object with one key whose value is not a list | that key is the root element | the same object |
| any other object | <root> holding the members |
{"root": {…}} |
list [a, b, …] |
<root><item>a</item><item>b</item>…</root> |
the same list (see the rule below) |
| list inside a list | <item> holding <item> elements |
{"item": [...]} per inner list (XML has no anonymous list) |
| empty list | <root/> (warning xml-empty-list) |
null |
| scalar / text | <root>text</root> |
{"root": "text"} |
| null | <root/> |
{"root": null} |
Reading rule that makes top-level lists round-trip: a root element named root whose only children are item
elements (no attribute, no text) is read as a list of those items. Consequences, all intended: [{"id":1},{"id":2}]
→ XML → JSON gives a list of two records again (values as text, as for every XML path); a one-entry list stays a
list; an object written by hand as {"item": [...]} also reads back as a list. The element names can be changed
(xml.root.name, xml.item.name); with other names the list is found under <root name>.<item name> when read back,
and the xml-root-wrapped warning says so. An empty list under a key is written as one empty element and reported
(xml-empty-list) instead of vanishing.
Before this fix a top-level list of two or more entries was written as several root elements (<root>…</root> <root>…</root>): not XML. The engine's own validation caught it, so json→xml, yaml→xml and plist→xml failed
on every such document. Tests: TreeTests.topLevelValuesToXML (27 cases: nine shapes × JSON, YAML, plist),
TreeTests.topLevelListRoundTripsThroughXML.
10. Known defects and caveats
- The official DuckDB library is a 62 MB dynamic library declaring macOS 11.0 as its minimum; it must be embedded and
signed with the app (
Contents/Frameworks). - An unterminated quote in a CSV is accepted by DuckDB (the field runs to the end of the file) rather than rejected.
- A zero-byte file with a SQLite extension is a valid empty database for SQLite: it inspects fine and
convertanswers "contains no ordinary table". - XLSX output with shared strings keeps up to 1 000 000 distinct strings / 256 MB in memory, then switches to inline strings; reading a workbook loads its shared-string table in memory (1 GB cap).
validatereads the output again: for large XLSX/ODS outputs this costs about as much as the conversion.- Preflight on very large documents (> 64 MB tree formats) is skipped and says so.
- Tree → XML: a list nested directly in a list, an empty list and a top-level scalar do not read back as what they
were (see §9b); an object whose only key is
itemunder a root namedrootreads back as a list. DataTests.cancellationStopsMidStream(CSV → Parquet case) is timing-sensitive: on a heavily loaded Mac (load average above 40 on 18 cores) the conversion once finished before the cancellation was seen and the test failed; it passed when re-run alone. The test, not the engine, depends on the load.
11. Tests
swift test from Packages/EngineData (debug), 2026-11-21, with the bundled LibreOffice present:
Test run with 70 tests in 6 suites passed after 55.780 seconds — 67 tests run (297 parameterised cases, among them
the 194 tabular and 25 tree paths and the 27 top-level XML shapes of §9b), 3 benchmark entries disabled unless
ORIX_DATA_BENCH is set.
| Suite | What it covers |
|---|---|
MatrixTests |
every declared tabular path: convert the canonical dataset (unicode incl. emoji and RTL, quotes, delimiters, line breaks, integers beyond 2^53, decimals, booleans, dates, timestamps, NULL vs empty string, leading-zero identifiers), run validate, read back and compare every cell |
TreeTests |
every declared tree path; JSON exactness (key order, big integers, decimal text, escapes, surrogates), YAML typing and quoting, TOML 1.0 features and rejections, XML mapping convention and its reversibility, XXE (file:// entity, parameter entity, UTF-16), billion laughs, YAML alias bomb, 100 000-level nesting, null policies, malformed documents |
SpreadsheetTests |
hand-assembled Excel-style packages (shared/inline/rich strings, phonetic runs, _xHHHH_ escapes, built-in and custom date formats, 1900 and 1904 systems, cached formula values, shared formulas, errors, merged cells, hidden sheets/rows/columns, prefixed elements, charts/images/pivots/comments/macros parts), number-format classification, writers' edge cases and loss tallies, sheet overflow (error and split), ODS typed cells and repeats, repeat bombs, ZIP reader (bomb limit, truncation, checksum, unzip -t), tokenizer, LibreOffice cross-check of our XLSX and ODS (conditional), XLS limits (conditional) |
TextTests |
delimiter/quote/header sniffing (semicolon + decimal comma, tab, pipe, quoted line breaks), encodings (UTF-8 ± BOM, UTF-16 LE/BE ± BOM, Latin-1, Windows-1252, Mac Roman, forced), conservative type detection and its full-file verification, forced types, NULL tokens, skipped rows, ragged rows, writing options, Excel preset, CSV-injection guard, column selection/renaming |
DataTests |
declared contract (path count, options, defaults), Arrow files written by Arrow C++ (plain, LZ4, stream), ZSTD refusal, damaged Arrow, byte-level IPC layout of our files, SQLite (types, views, triggers, hostile names, read-only hardening), DuckDB files (all types through every streamed target), Parquet codecs/row groups/metadata and broken footers, nested JSON (flatten/json/native, record paths), damaged inputs for every reader, hostile file/table/column names, the DuckDB sandbox, memory limit and spilling, cancellation mid-stream for five targets, preview API, validation catching wrong outputs |