Repository Public License
1# Data engine (`data`) — spreadsheets, tables and structured documents23Package `Packages/EngineData` (module `EngineData`, factory `makeEngines()`), engine id `data`, display name4"Spreadsheets & Data". Everything below was implemented, run and measured on 2026-11-21 on this Mac5(Apple M-series, 18 cores, 48 GB, macOS 27.0, Swift 6.4.2). Nothing is listed that is not covered by a passing test.67Formats: XLSX, XLS, ODS, CSV, TSV, JSON, JSON Lines, Parquet, Arrow IPC (file = Feather v2, and stream), SQLite,8DuckDB database files, XML, YAML, TOML, property lists (XML and binary).910## 1. Architecture in one page1112- **DuckDB v2.0.1** (official `libduckdb`, arm64, C API) is the tabular hub. Every tabular conversion is13 "source → DuckDB relation → target":14 - CSV/TSV, JSON/JSON Lines, Parquet and DuckDB files are scanned by DuckDB itself (streaming, multi-threaded,15 bounded by `memory_limit`, spilling to a `temp_directory` inside the job's work directory).16 - XLSX, ODS, XLS, SQLite and Arrow are read by the engine's own readers and staged, row by row, into a DuckDB17 table that lives in a database file inside the work directory (so they are not limited by memory either). Each18 column records the kinds of values actually seen and is cast to the narrowest type that holds them all exactly.19 - XML, YAML, TOML, property lists and JSON objects are parsed into an ordered value tree; the list of records is20 located and handed to DuckDB.21 - CSV/TSV/JSON/JSON Lines/Parquet are written by DuckDB `COPY`, DuckDB files by `CREATE TABLE AS`; everything else22 is written by the engine's own writers from streamed result chunks.23- **Tree conversions** (JSON, YAML, TOML, XML, plist ↔ each other) never go through a table: parse → ordered tree →24 serialise.25- **Sandbox.** Each job opens its own DuckDB session with `autoinstall_known_extensions=false`,26 `autoload_known_extensions=false`, `enable_external_access=false`, `allowed_directories` = the job's private27 workspace + the work directory, `allowed_paths` = the job's source file, `TimeZone='UTC'`, then28 `lock_configuration=true`. SQL can therefore not read or write any other file, cannot install or load an extension29 and cannot undo these settings (test `duckdbIsLockedDown`: 23 escape attempts, all refused). No user text is ever30 spliced into SQL unquoted: names and paths go through `SQL.ident` / `SQL.literal`31 (test `hostileFileSheetAndColumnNames`). A source file whose name contains glob characters (`* ? [ ] { }`) is reached32 through a hard link with a neutral name, so the name can never match another file.33- **What the bundled DuckDB build contains** (verified by running it): core functions, CSV, `parquet`, `json`, `icu`.34 Its SQLite, Excel, Arrow and spatial support live in downloadable extensions and are **not used**.35- The pipeline is synchronous and runs on a dedicated 8 MB-stack thread per job; task cancellation interrupts the36 running DuckDB statement / SQLite statement / LibreOffice process and is checked in every row loop.37- **Progress** is measured: DuckDB's own scan progress for `COPY`/verification passes, compressed bytes consumed for38 XLSX/ODS, rows written over rows staged for streamed targets, file offset for Arrow. A job declares how many full39 passes over the data it makes (1 or 2) and reports (finished passes + fraction of the current one) / passes. Steps40 that cannot be measured (LibreOffice, tree documents) report `fraction: nil`.4142### Bundled / used components4344| Component | Version | License | How | Size |45|---|---|---|---|---|46| DuckDB `libduckdb.dylib` | v2.0.1 (pinned, SHA-256 in `EmbeddedEngines/duckdb/build.sh`) | MIT | dynamic library in `CDuckDB.xcframework`, SwiftPM `binaryTarget` | 62 MB (arm64) |47| Yams | 6.2.2 (exact pin) | MIT | SwiftPM source dependency (wraps libyaml, compiled by SwiftPM) | source |48| SQLite | system `libsqlite3` (3.54.0 on this Mac) | public domain | linked from the macOS SDK | — |49| Compression, zlib (`crc32` only) | system | — | Apple frameworks / SDK | — |50| LibreOffice (`soffice`) | bundled by the documents engine in `EmbeddedEngines/tools/LibreOffice.app` | MPL-2.0 | separate process through `ProcessRunner`, **XLS only** | not owned by this engine |5152Written for this engine (no third-party code): ZIP reader/writer, streaming XML tokenizer, XLSX reader/writer,53ODS reader/writer, Arrow IPC reader/writer (FlatBuffers metadata + LZ4-frame decoder), exact JSON parser/serialiser,54TOML 1.0 parser/serialiser, XML ↔ tree mapping, text-encoding detection/transcoding, delimiter sniffer, column type55detection.5657## 2. Declared paths5859219 paths are declared when the bundled LibreOffice is present (as today), 191 otherwise: the 28 paths that60read or write XLS exist only with it (`paths` is computed; `convert` answers `engineUnavailable` if XLS is requested61without it). No named operations; every path is a plain format conversion. Same-format paths exist only where62re-writing with other options is meaningful and safe: CSV, TSV, JSON Lines, Parquet, JSON, YAML, TOML, XML, plist.6364Status is `supported` for all of them: each one converts a fixture, passes the engine's own `validate`, and is read65back and compared value by value in the parameterised tests `MatrixTests.path` (194 tabular paths) and66`TreeTests.path` (25 tree paths).6768**Fidelity as declared**6970- `lossless`: Parquet → Parquet, Parquet → DuckDB, JSON → JSON, JSON → YAML, plist → plist. The output reads back71 identical (types and values; for plist, the same property list).72- `partial`: every path from a workbook (XLSX/XLS/ODS: only cell values are carried), and every path from a73 database file to a single-table target (one table per output file).74- `structural`: everything else — the content is kept but re-expressed (types detected from text, dates as ISO text,75 nested values flattened or stringified…). On these paths the canonical dataset of the tests still reads back with76 **equal values** (compared as numbers / instants / text); the differences are type re-expression, documented below.7778<details><summary>Full path table (219 rows)</summary>7980| Source | Destination | Operation | Status | Fidelity | Preserves | Backend (read → write) | Test |81|---|---|---|---|---|---|---|---|82| xlsx | xls | — | supported | partial | multipleSheets, tables | engine XLSX reader → staging → engine XLSX writer → LibreOffice | `MatrixTests.path` (xlsx → xls) |83| xlsx | ods | — | supported | partial | multipleSheets, tables | engine XLSX reader → staging → engine ODS writer | `MatrixTests.path` (xlsx → ods) |84| xlsx | csv | — | supported | partial | tables | engine XLSX reader → staging → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (xlsx → csv) |85| xlsx | tsv | — | supported | partial | tables | engine XLSX reader → staging → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (xlsx → tsv) |86| xlsx | json | — | supported | partial | tables | engine XLSX reader → staging → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (xlsx → json) |87| xlsx | jsonl | — | supported | partial | tables | engine XLSX reader → staging → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (xlsx → jsonl) |88| xlsx | parquet | — | supported | partial | tables | engine XLSX reader → staging → DuckDB `COPY … (FORMAT parquet)` | `MatrixTests.path` (xlsx → parquet) |89| xlsx | arrow | — | supported | partial | tables | engine XLSX reader → staging → engine Arrow IPC writer | `MatrixTests.path` (xlsx → arrow) |90| xlsx | sqlite | — | supported | partial | multipleSheets, tables | engine XLSX reader → staging → system libsqlite3 | `MatrixTests.path` (xlsx → sqlite) |91| xlsx | duckdb | — | supported | partial | multipleSheets, tables | engine XLSX reader → staging → DuckDB `CREATE TABLE AS` | `MatrixTests.path` (xlsx → duckdb) |92| xlsx | xml | — | supported | partial | tables | engine XLSX reader → staging → engine XML writer | `MatrixTests.path` (xlsx → xml) |93| xlsx | yaml | — | supported | partial | tables | engine XLSX reader → staging → Yams (row by row) | `MatrixTests.path` (xlsx → yaml) |94| xlsx | toml | — | supported | partial | tables | engine XLSX reader → staging → engine TOML writer | `MatrixTests.path` (xlsx → toml) |95| xlsx | plist | — | supported | partial | tables | engine XLSX reader → staging → PropertyListSerialization | `MatrixTests.path` (xlsx → plist) |96| xls | xlsx | — | supported | partial | multipleSheets, tables | LibreOffice → XLSX → engine XLSX reader → engine XLSX writer | `MatrixTests.path` (xls → xlsx) |97| xls | ods | — | supported | partial | multipleSheets, tables | LibreOffice → XLSX → engine XLSX reader → engine ODS writer | `MatrixTests.path` (xls → ods) |98| xls | csv | — | supported | partial | tables | LibreOffice → XLSX → engine XLSX reader → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (xls → csv) |99| xls | tsv | — | supported | partial | tables | LibreOffice → XLSX → engine XLSX reader → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (xls → tsv) |100| xls | json | — | supported | partial | tables | LibreOffice → XLSX → engine XLSX reader → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (xls → json) |101| xls | jsonl | — | supported | partial | tables | LibreOffice → XLSX → engine XLSX reader → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (xls → jsonl) |102| xls | parquet | — | supported | partial | tables | LibreOffice → XLSX → engine XLSX reader → DuckDB `COPY … (FORMAT parquet)` | `MatrixTests.path` (xls → parquet) |103| xls | arrow | — | supported | partial | tables | LibreOffice → XLSX → engine XLSX reader → engine Arrow IPC writer | `MatrixTests.path` (xls → arrow) |104| xls | sqlite | — | supported | partial | multipleSheets, tables | LibreOffice → XLSX → engine XLSX reader → system libsqlite3 | `MatrixTests.path` (xls → sqlite) |105| xls | duckdb | — | supported | partial | multipleSheets, tables | LibreOffice → XLSX → engine XLSX reader → DuckDB `CREATE TABLE AS` | `MatrixTests.path` (xls → duckdb) |106| xls | xml | — | supported | partial | tables | LibreOffice → XLSX → engine XLSX reader → engine XML writer | `MatrixTests.path` (xls → xml) |107| xls | yaml | — | supported | partial | tables | LibreOffice → XLSX → engine XLSX reader → Yams (row by row) | `MatrixTests.path` (xls → yaml) |108| xls | toml | — | supported | partial | tables | LibreOffice → XLSX → engine XLSX reader → engine TOML writer | `MatrixTests.path` (xls → toml) |109| xls | plist | — | supported | partial | tables | LibreOffice → XLSX → engine XLSX reader → PropertyListSerialization | `MatrixTests.path` (xls → plist) |110| ods | xlsx | — | supported | partial | multipleSheets, tables | engine ODS reader → staging → engine XLSX writer | `MatrixTests.path` (ods → xlsx) |111| ods | xls | — | supported | partial | multipleSheets, tables | engine ODS reader → staging → engine XLSX writer → LibreOffice | `MatrixTests.path` (ods → xls) |112| ods | csv | — | supported | partial | tables | engine ODS reader → staging → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (ods → csv) |113| ods | tsv | — | supported | partial | tables | engine ODS reader → staging → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (ods → tsv) |114| ods | json | — | supported | partial | tables | engine ODS reader → staging → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (ods → json) |115| ods | jsonl | — | supported | partial | tables | engine ODS reader → staging → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (ods → jsonl) |116| ods | parquet | — | supported | partial | tables | engine ODS reader → staging → DuckDB `COPY … (FORMAT parquet)` | `MatrixTests.path` (ods → parquet) |117| ods | arrow | — | supported | partial | tables | engine ODS reader → staging → engine Arrow IPC writer | `MatrixTests.path` (ods → arrow) |118| ods | sqlite | — | supported | partial | multipleSheets, tables | engine ODS reader → staging → system libsqlite3 | `MatrixTests.path` (ods → sqlite) |119| ods | duckdb | — | supported | partial | multipleSheets, tables | engine ODS reader → staging → DuckDB `CREATE TABLE AS` | `MatrixTests.path` (ods → duckdb) |120| ods | xml | — | supported | partial | tables | engine ODS reader → staging → engine XML writer | `MatrixTests.path` (ods → xml) |121| ods | yaml | — | supported | partial | tables | engine ODS reader → staging → Yams (row by row) | `MatrixTests.path` (ods → yaml) |122| ods | toml | — | supported | partial | tables | engine ODS reader → staging → engine TOML writer | `MatrixTests.path` (ods → toml) |123| ods | plist | — | supported | partial | tables | engine ODS reader → staging → PropertyListSerialization | `MatrixTests.path` (ods → plist) |124| csv | xlsx | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → engine XLSX writer | `MatrixTests.path` (csv → xlsx) |125| csv | xls | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → engine XLSX writer → LibreOffice | `MatrixTests.path` (csv → xls) |126| csv | ods | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → engine ODS writer | `MatrixTests.path` (csv → ods) |127| csv | csv | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (csv → csv) |128| csv | tsv | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (csv → tsv) |129| csv | json | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (csv → json) |130| csv | jsonl | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (csv → jsonl) |131| csv | parquet | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → DuckDB `COPY … (FORMAT parquet)` | `MatrixTests.path` (csv → parquet) |132| csv | arrow | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → engine Arrow IPC writer | `MatrixTests.path` (csv → arrow) |133| csv | sqlite | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → system libsqlite3 | `MatrixTests.path` (csv → sqlite) |134| csv | duckdb | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → DuckDB `CREATE TABLE AS` | `MatrixTests.path` (csv → duckdb) |135| csv | xml | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → engine XML writer | `MatrixTests.path` (csv → xml) |136| csv | yaml | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → Yams (row by row) | `MatrixTests.path` (csv → yaml) |137| csv | toml | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → engine TOML writer | `MatrixTests.path` (csv → toml) |138| csv | plist | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → PropertyListSerialization | `MatrixTests.path` (csv → plist) |139| tsv | xlsx | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → engine XLSX writer | `MatrixTests.path` (tsv → xlsx) |140| tsv | xls | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → engine XLSX writer → LibreOffice | `MatrixTests.path` (tsv → xls) |141| tsv | ods | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → engine ODS writer | `MatrixTests.path` (tsv → ods) |142| tsv | csv | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (tsv → csv) |143| tsv | tsv | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (tsv → tsv) |144| tsv | json | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (tsv → json) |145| tsv | jsonl | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (tsv → jsonl) |146| tsv | parquet | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → DuckDB `COPY … (FORMAT parquet)` | `MatrixTests.path` (tsv → parquet) |147| tsv | arrow | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → engine Arrow IPC writer | `MatrixTests.path` (tsv → arrow) |148| tsv | sqlite | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → system libsqlite3 | `MatrixTests.path` (tsv → sqlite) |149| tsv | duckdb | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → DuckDB `CREATE TABLE AS` | `MatrixTests.path` (tsv → duckdb) |150| tsv | xml | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → engine XML writer | `MatrixTests.path` (tsv → xml) |151| tsv | yaml | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → Yams (row by row) | `MatrixTests.path` (tsv → yaml) |152| tsv | toml | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → engine TOML writer | `MatrixTests.path` (tsv → toml) |153| tsv | plist | — | supported | structural | tables | DuckDB `read_csv` + engine type detection → PropertyListSerialization | `MatrixTests.path` (tsv → plist) |154| json | xlsx | — | supported | structural | tables | DuckDB `read_json` (top-level array) / engine JSON parser (object) → engine XLSX writer | `MatrixTests.path` (json → xlsx) |155| json | xls | — | supported | structural | tables | DuckDB `read_json` (top-level array) / engine JSON parser (object) → engine XLSX writer → LibreOffice | `MatrixTests.path` (json → xls) |156| json | ods | — | supported | structural | tables | DuckDB `read_json` (top-level array) / engine JSON parser (object) → engine ODS writer | `MatrixTests.path` (json → ods) |157| json | csv | — | supported | structural | tables | DuckDB `read_json` (top-level array) / engine JSON parser (object) → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (json → csv) |158| json | tsv | — | supported | structural | tables | DuckDB `read_json` (top-level array) / engine JSON parser (object) → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (json → tsv) |159| json | json | — | supported | lossless | | engine JSON codec → engine JSON codec | `TreeTests.path` (json → json) |160| json | jsonl | — | supported | structural | tables | DuckDB `read_json` (top-level array) / engine JSON parser (object) → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (json → jsonl) |161| json | parquet | — | supported | structural | tables | DuckDB `read_json` (top-level array) / engine JSON parser (object) → DuckDB `COPY … (FORMAT parquet)` | `MatrixTests.path` (json → parquet) |162| json | arrow | — | supported | structural | tables | DuckDB `read_json` (top-level array) / engine JSON parser (object) → engine Arrow IPC writer | `MatrixTests.path` (json → arrow) |163| json | sqlite | — | supported | structural | tables | DuckDB `read_json` (top-level array) / engine JSON parser (object) → system libsqlite3 | `MatrixTests.path` (json → sqlite) |164| json | duckdb | — | supported | structural | tables | DuckDB `read_json` (top-level array) / engine JSON parser (object) → DuckDB `CREATE TABLE AS` | `MatrixTests.path` (json → duckdb) |165| json | xml | — | supported | structural | | engine JSON codec → XMLParser / engine XML writer | `TreeTests.path` (json → xml) |166| json | yaml | — | supported | lossless | | engine JSON codec → Yams | `TreeTests.path` (json → yaml) |167| json | toml | — | supported | structural | | engine JSON codec → engine TOML codec | `TreeTests.path` (json → toml) |168| json | plist | — | supported | structural | | engine JSON codec → PropertyListSerialization | `TreeTests.path` (json → plist) |169| jsonl | xlsx | — | supported | structural | tables | DuckDB `read_json` → engine XLSX writer | `MatrixTests.path` (jsonl → xlsx) |170| jsonl | xls | — | supported | structural | tables | DuckDB `read_json` → engine XLSX writer → LibreOffice | `MatrixTests.path` (jsonl → xls) |171| jsonl | ods | — | supported | structural | tables | DuckDB `read_json` → engine ODS writer | `MatrixTests.path` (jsonl → ods) |172| jsonl | csv | — | supported | structural | tables | DuckDB `read_json` → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (jsonl → csv) |173| jsonl | tsv | — | supported | structural | tables | DuckDB `read_json` → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (jsonl → tsv) |174| jsonl | json | — | supported | structural | tables | DuckDB `read_json` → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (jsonl → json) |175| jsonl | jsonl | — | supported | structural | tables | DuckDB `read_json` → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (jsonl → jsonl) |176| jsonl | parquet | — | supported | structural | tables | DuckDB `read_json` → DuckDB `COPY … (FORMAT parquet)` | `MatrixTests.path` (jsonl → parquet) |177| jsonl | arrow | — | supported | structural | tables | DuckDB `read_json` → engine Arrow IPC writer | `MatrixTests.path` (jsonl → arrow) |178| jsonl | sqlite | — | supported | structural | tables | DuckDB `read_json` → system libsqlite3 | `MatrixTests.path` (jsonl → sqlite) |179| jsonl | duckdb | — | supported | structural | tables | DuckDB `read_json` → DuckDB `CREATE TABLE AS` | `MatrixTests.path` (jsonl → duckdb) |180| jsonl | xml | — | supported | structural | tables | DuckDB `read_json` → engine XML writer | `MatrixTests.path` (jsonl → xml) |181| jsonl | yaml | — | supported | structural | tables | DuckDB `read_json` → Yams (row by row) | `MatrixTests.path` (jsonl → yaml) |182| jsonl | toml | — | supported | structural | tables | DuckDB `read_json` → engine TOML writer | `MatrixTests.path` (jsonl → toml) |183| jsonl | plist | — | supported | structural | tables | DuckDB `read_json` → PropertyListSerialization | `MatrixTests.path` (jsonl → plist) |184| parquet | xlsx | — | supported | structural | tables | DuckDB `read_parquet` → engine XLSX writer | `MatrixTests.path` (parquet → xlsx) |185| parquet | xls | — | supported | structural | tables | DuckDB `read_parquet` → engine XLSX writer → LibreOffice | `MatrixTests.path` (parquet → xls) |186| parquet | ods | — | supported | structural | tables | DuckDB `read_parquet` → engine ODS writer | `MatrixTests.path` (parquet → ods) |187| parquet | csv | — | supported | structural | tables | DuckDB `read_parquet` → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (parquet → csv) |188| parquet | tsv | — | supported | structural | tables | DuckDB `read_parquet` → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (parquet → tsv) |189| parquet | json | — | supported | structural | tables | DuckDB `read_parquet` → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (parquet → json) |190| parquet | jsonl | — | supported | structural | tables | DuckDB `read_parquet` → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (parquet → jsonl) |191| parquet | parquet | — | supported | lossless | tables | DuckDB `read_parquet` → DuckDB `COPY … (FORMAT parquet)` | `MatrixTests.path` (parquet → parquet) |192| parquet | arrow | — | supported | structural | tables | DuckDB `read_parquet` → engine Arrow IPC writer | `MatrixTests.path` (parquet → arrow) |193| parquet | sqlite | — | supported | structural | tables | DuckDB `read_parquet` → system libsqlite3 | `MatrixTests.path` (parquet → sqlite) |194| parquet | duckdb | — | supported | lossless | tables | DuckDB `read_parquet` → DuckDB `CREATE TABLE AS` | `MatrixTests.path` (parquet → duckdb) |195| parquet | xml | — | supported | structural | tables | DuckDB `read_parquet` → engine XML writer | `MatrixTests.path` (parquet → xml) |196| parquet | yaml | — | supported | structural | tables | DuckDB `read_parquet` → Yams (row by row) | `MatrixTests.path` (parquet → yaml) |197| parquet | toml | — | supported | structural | tables | DuckDB `read_parquet` → engine TOML writer | `MatrixTests.path` (parquet → toml) |198| parquet | plist | — | supported | structural | tables | DuckDB `read_parquet` → PropertyListSerialization | `MatrixTests.path` (parquet → plist) |199| arrow | xlsx | — | supported | structural | tables | engine Arrow IPC reader → staging → engine XLSX writer | `MatrixTests.path` (arrow → xlsx) |200| arrow | xls | — | supported | structural | tables | engine Arrow IPC reader → staging → engine XLSX writer → LibreOffice | `MatrixTests.path` (arrow → xls) |201| arrow | ods | — | supported | structural | tables | engine Arrow IPC reader → staging → engine ODS writer | `MatrixTests.path` (arrow → ods) |202| arrow | csv | — | supported | structural | tables | engine Arrow IPC reader → staging → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (arrow → csv) |203| arrow | tsv | — | supported | structural | tables | engine Arrow IPC reader → staging → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (arrow → tsv) |204| arrow | json | — | supported | structural | tables | engine Arrow IPC reader → staging → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (arrow → json) |205| arrow | jsonl | — | supported | structural | tables | engine Arrow IPC reader → staging → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (arrow → jsonl) |206| arrow | parquet | — | supported | structural | tables | engine Arrow IPC reader → staging → DuckDB `COPY … (FORMAT parquet)` | `MatrixTests.path` (arrow → parquet) |207| arrow | sqlite | — | supported | structural | tables | engine Arrow IPC reader → staging → system libsqlite3 | `MatrixTests.path` (arrow → sqlite) |208| arrow | duckdb | — | supported | structural | tables | engine Arrow IPC reader → staging → DuckDB `CREATE TABLE AS` | `MatrixTests.path` (arrow → duckdb) |209| arrow | xml | — | supported | structural | tables | engine Arrow IPC reader → staging → engine XML writer | `MatrixTests.path` (arrow → xml) |210| arrow | yaml | — | supported | structural | tables | engine Arrow IPC reader → staging → Yams (row by row) | `MatrixTests.path` (arrow → yaml) |211| arrow | toml | — | supported | structural | tables | engine Arrow IPC reader → staging → engine TOML writer | `MatrixTests.path` (arrow → toml) |212| arrow | plist | — | supported | structural | tables | engine Arrow IPC reader → staging → PropertyListSerialization | `MatrixTests.path` (arrow → plist) |213| sqlite | xlsx | — | supported | structural | multipleSheets, tables | system libsqlite3 → staging → engine XLSX writer | `MatrixTests.path` (sqlite → xlsx) |214| sqlite | xls | — | supported | structural | multipleSheets, tables | system libsqlite3 → staging → engine XLSX writer → LibreOffice | `MatrixTests.path` (sqlite → xls) |215| sqlite | ods | — | supported | structural | multipleSheets, tables | system libsqlite3 → staging → engine ODS writer | `MatrixTests.path` (sqlite → ods) |216| sqlite | csv | — | supported | partial | tables | system libsqlite3 → staging → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (sqlite → csv) |217| sqlite | tsv | — | supported | partial | tables | system libsqlite3 → staging → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (sqlite → tsv) |218| sqlite | json | — | supported | partial | tables | system libsqlite3 → staging → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (sqlite → json) |219| sqlite | jsonl | — | supported | partial | tables | system libsqlite3 → staging → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (sqlite → jsonl) |220| sqlite | parquet | — | supported | partial | tables | system libsqlite3 → staging → DuckDB `COPY … (FORMAT parquet)` | `MatrixTests.path` (sqlite → parquet) |221| sqlite | arrow | — | supported | partial | tables | system libsqlite3 → staging → engine Arrow IPC writer | `MatrixTests.path` (sqlite → arrow) |222| sqlite | duckdb | — | supported | structural | multipleSheets, tables | system libsqlite3 → staging → DuckDB `CREATE TABLE AS` | `MatrixTests.path` (sqlite → duckdb) |223| sqlite | xml | — | supported | partial | tables | system libsqlite3 → staging → engine XML writer | `MatrixTests.path` (sqlite → xml) |224| sqlite | yaml | — | supported | partial | tables | system libsqlite3 → staging → Yams (row by row) | `MatrixTests.path` (sqlite → yaml) |225| sqlite | toml | — | supported | partial | tables | system libsqlite3 → staging → engine TOML writer | `MatrixTests.path` (sqlite → toml) |226| sqlite | plist | — | supported | partial | tables | system libsqlite3 → staging → PropertyListSerialization | `MatrixTests.path` (sqlite → plist) |227| duckdb | xlsx | — | supported | structural | multipleSheets, tables | DuckDB `ATTACH` (read-only) → engine XLSX writer | `MatrixTests.path` (duckdb → xlsx) |228| duckdb | xls | — | supported | structural | multipleSheets, tables | DuckDB `ATTACH` (read-only) → engine XLSX writer → LibreOffice | `MatrixTests.path` (duckdb → xls) |229| duckdb | ods | — | supported | structural | multipleSheets, tables | DuckDB `ATTACH` (read-only) → engine ODS writer | `MatrixTests.path` (duckdb → ods) |230| duckdb | csv | — | supported | partial | tables | DuckDB `ATTACH` (read-only) → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (duckdb → csv) |231| duckdb | tsv | — | supported | partial | tables | DuckDB `ATTACH` (read-only) → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (duckdb → tsv) |232| duckdb | json | — | supported | partial | tables | DuckDB `ATTACH` (read-only) → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (duckdb → json) |233| duckdb | jsonl | — | supported | partial | tables | DuckDB `ATTACH` (read-only) → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (duckdb → jsonl) |234| duckdb | parquet | — | supported | partial | tables | DuckDB `ATTACH` (read-only) → DuckDB `COPY … (FORMAT parquet)` | `MatrixTests.path` (duckdb → parquet) |235| duckdb | arrow | — | supported | partial | tables | DuckDB `ATTACH` (read-only) → engine Arrow IPC writer | `MatrixTests.path` (duckdb → arrow) |236| duckdb | sqlite | — | supported | structural | multipleSheets, tables | DuckDB `ATTACH` (read-only) → system libsqlite3 | `MatrixTests.path` (duckdb → sqlite) |237| duckdb | xml | — | supported | partial | tables | DuckDB `ATTACH` (read-only) → engine XML writer | `MatrixTests.path` (duckdb → xml) |238| duckdb | yaml | — | supported | partial | tables | DuckDB `ATTACH` (read-only) → Yams (row by row) | `MatrixTests.path` (duckdb → yaml) |239| duckdb | toml | — | supported | partial | tables | DuckDB `ATTACH` (read-only) → engine TOML writer | `MatrixTests.path` (duckdb → toml) |240| duckdb | plist | — | supported | partial | tables | DuckDB `ATTACH` (read-only) → PropertyListSerialization | `MatrixTests.path` (duckdb → plist) |241| xml | xlsx | — | supported | structural | tables | XMLParser → records → engine XLSX writer | `MatrixTests.path` (xml → xlsx) |242| xml | xls | — | supported | structural | tables | XMLParser → records → engine XLSX writer → LibreOffice | `MatrixTests.path` (xml → xls) |243| xml | ods | — | supported | structural | tables | XMLParser → records → engine ODS writer | `MatrixTests.path` (xml → ods) |244| xml | csv | — | supported | structural | tables | XMLParser → records → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (xml → csv) |245| xml | tsv | — | supported | structural | tables | XMLParser → records → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (xml → tsv) |246| xml | json | — | supported | structural | | XMLParser / engine XML writer → engine JSON codec | `TreeTests.path` (xml → json) |247| xml | jsonl | — | supported | structural | tables | XMLParser → records → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (xml → jsonl) |248| xml | parquet | — | supported | structural | tables | XMLParser → records → DuckDB `COPY … (FORMAT parquet)` | `MatrixTests.path` (xml → parquet) |249| xml | arrow | — | supported | structural | tables | XMLParser → records → engine Arrow IPC writer | `MatrixTests.path` (xml → arrow) |250| xml | sqlite | — | supported | structural | tables | XMLParser → records → system libsqlite3 | `MatrixTests.path` (xml → sqlite) |251| xml | duckdb | — | supported | structural | tables | XMLParser → records → DuckDB `CREATE TABLE AS` | `MatrixTests.path` (xml → duckdb) |252| xml | xml | — | supported | structural | | XMLParser / engine XML writer → XMLParser / engine XML writer | `TreeTests.path` (xml → xml) |253| xml | yaml | — | supported | structural | | XMLParser / engine XML writer → Yams | `TreeTests.path` (xml → yaml) |254| xml | toml | — | supported | structural | | XMLParser / engine XML writer → engine TOML codec | `TreeTests.path` (xml → toml) |255| xml | plist | — | supported | structural | | XMLParser / engine XML writer → PropertyListSerialization | `TreeTests.path` (xml → plist) |256| yaml | xlsx | — | supported | structural | tables | Yams → records → engine XLSX writer | `MatrixTests.path` (yaml → xlsx) |257| yaml | xls | — | supported | structural | tables | Yams → records → engine XLSX writer → LibreOffice | `MatrixTests.path` (yaml → xls) |258| yaml | ods | — | supported | structural | tables | Yams → records → engine ODS writer | `MatrixTests.path` (yaml → ods) |259| yaml | csv | — | supported | structural | tables | Yams → records → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (yaml → csv) |260| yaml | tsv | — | supported | structural | tables | Yams → records → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (yaml → tsv) |261| yaml | json | — | supported | structural | | Yams → engine JSON codec | `TreeTests.path` (yaml → json) |262| yaml | jsonl | — | supported | structural | tables | Yams → records → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (yaml → jsonl) |263| yaml | parquet | — | supported | structural | tables | Yams → records → DuckDB `COPY … (FORMAT parquet)` | `MatrixTests.path` (yaml → parquet) |264| yaml | arrow | — | supported | structural | tables | Yams → records → engine Arrow IPC writer | `MatrixTests.path` (yaml → arrow) |265| yaml | sqlite | — | supported | structural | tables | Yams → records → system libsqlite3 | `MatrixTests.path` (yaml → sqlite) |266| yaml | duckdb | — | supported | structural | tables | Yams → records → DuckDB `CREATE TABLE AS` | `MatrixTests.path` (yaml → duckdb) |267| yaml | xml | — | supported | structural | | Yams → XMLParser / engine XML writer | `TreeTests.path` (yaml → xml) |268| yaml | yaml | — | supported | structural | | Yams → Yams | `TreeTests.path` (yaml → yaml) |269| yaml | toml | — | supported | structural | | Yams → engine TOML codec | `TreeTests.path` (yaml → toml) |270| yaml | plist | — | supported | structural | | Yams → PropertyListSerialization | `TreeTests.path` (yaml → plist) |271| toml | xlsx | — | supported | structural | tables | engine TOML parser → records → engine XLSX writer | `MatrixTests.path` (toml → xlsx) |272| toml | xls | — | supported | structural | tables | engine TOML parser → records → engine XLSX writer → LibreOffice | `MatrixTests.path` (toml → xls) |273| toml | ods | — | supported | structural | tables | engine TOML parser → records → engine ODS writer | `MatrixTests.path` (toml → ods) |274| toml | csv | — | supported | structural | tables | engine TOML parser → records → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (toml → csv) |275| toml | tsv | — | supported | structural | tables | engine TOML parser → records → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (toml → tsv) |276| toml | json | — | supported | structural | | engine TOML codec → engine JSON codec | `TreeTests.path` (toml → json) |277| toml | jsonl | — | supported | structural | tables | engine TOML parser → records → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (toml → jsonl) |278| toml | parquet | — | supported | structural | tables | engine TOML parser → records → DuckDB `COPY … (FORMAT parquet)` | `MatrixTests.path` (toml → parquet) |279| toml | arrow | — | supported | structural | tables | engine TOML parser → records → engine Arrow IPC writer | `MatrixTests.path` (toml → arrow) |280| toml | sqlite | — | supported | structural | tables | engine TOML parser → records → system libsqlite3 | `MatrixTests.path` (toml → sqlite) |281| toml | duckdb | — | supported | structural | tables | engine TOML parser → records → DuckDB `CREATE TABLE AS` | `MatrixTests.path` (toml → duckdb) |282| toml | xml | — | supported | structural | | engine TOML codec → XMLParser / engine XML writer | `TreeTests.path` (toml → xml) |283| toml | yaml | — | supported | structural | | engine TOML codec → Yams | `TreeTests.path` (toml → yaml) |284| toml | toml | — | supported | structural | | engine TOML codec → engine TOML codec | `TreeTests.path` (toml → toml) |285| toml | plist | — | supported | structural | | engine TOML codec → PropertyListSerialization | `TreeTests.path` (toml → plist) |286| plist | xlsx | — | supported | structural | tables | PropertyListSerialization → records → engine XLSX writer | `MatrixTests.path` (plist → xlsx) |287| plist | xls | — | supported | structural | tables | PropertyListSerialization → records → engine XLSX writer → LibreOffice | `MatrixTests.path` (plist → xls) |288| plist | ods | — | supported | structural | tables | PropertyListSerialization → records → engine ODS writer | `MatrixTests.path` (plist → ods) |289| plist | csv | — | supported | structural | tables | PropertyListSerialization → records → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (plist → csv) |290| plist | tsv | — | supported | structural | tables | PropertyListSerialization → records → DuckDB `COPY … (FORMAT csv)` | `MatrixTests.path` (plist → tsv) |291| plist | json | — | supported | structural | | PropertyListSerialization → engine JSON codec | `TreeTests.path` (plist → json) |292| plist | jsonl | — | supported | structural | tables | PropertyListSerialization → records → DuckDB `COPY … (FORMAT json)` | `MatrixTests.path` (plist → jsonl) |293| plist | parquet | — | supported | structural | tables | PropertyListSerialization → records → DuckDB `COPY … (FORMAT parquet)` | `MatrixTests.path` (plist → parquet) |294| plist | arrow | — | supported | structural | tables | PropertyListSerialization → records → engine Arrow IPC writer | `MatrixTests.path` (plist → arrow) |295| plist | sqlite | — | supported | structural | tables | PropertyListSerialization → records → system libsqlite3 | `MatrixTests.path` (plist → sqlite) |296| plist | duckdb | — | supported | structural | tables | PropertyListSerialization → records → DuckDB `CREATE TABLE AS` | `MatrixTests.path` (plist → duckdb) |297| plist | xml | — | supported | structural | | PropertyListSerialization → XMLParser / engine XML writer | `TreeTests.path` (plist → xml) |298| plist | yaml | — | supported | structural | | PropertyListSerialization → Yams | `TreeTests.path` (plist → yaml) |299| plist | toml | — | supported | structural | | PropertyListSerialization → engine TOML codec | `TreeTests.path` (plist → toml) |300| plist | plist | — | supported | lossless | | PropertyListSerialization → PropertyListSerialization | `TreeTests.path` (plist → plist) |301302</details>303304### Limitations shown to the user (verbatim from the engine, with the paths they apply to)305306- Only cell values are converted. Formulas (their last calculated values are kept), styles, number formats, column widths, charts, images, pivot tables, comments, data validation and conditional formatting are not carried over. — *from ods, xls, xlsx (42 paths)*307- Merged cells keep their value in the first cell only. — *from ods, xls, xlsx (42 paths)*308- Spreadsheet numbers are 64-bit floats: integers beyond 2^53 and long decimals are written as text; NaN and infinities become empty cells; time-zoned instants are written as UTC wall-clock times. — *to ods, xls, xlsx (42 paths)*309- Date-times are stored to the millisecond. — *to xls, xlsx (28 paths)*310- At most 65,536 rows and 256 columns per sheet (legacy format limit); produced through the bundled LibreOffice. — *to xls (14 paths)*311- At most 1,048,576 rows and 16,384 columns per sheet (larger tables stop with an error, or are split across sheets on request). — *to ods, xlsx (28 paths)*312- One sheet per output file: the first one by default, a chosen one, or all of them as separate files. — *from ods, xls, xlsx (30 paths)*313- Text formats carry no types: dates are written in ISO 8601, binary values as Base64, nested values as JSON text. — *to csv, tsv (30 paths)*314- JSON has no date, binary, NaN or infinity: dates become ISO 8601 text, binary values Base64 text, NaN and infinities null. — *to json, jsonl (25 paths)*315- Columns are written as int64, float64, bool, utf8, binary, date32, time64[us], timestamp[us] and decimal128; nested values as JSON text. Buffers are not compressed. — *to arrow (14 paths)*316- SQLite numbers are 64-bit: exact decimals and 128-bit integers are stored as TEXT; dates and times as ISO 8601 text in DATE / DATETIME / TIME columns. — *to sqlite (14 paths)*317- XML carries text only; empty values are omitted, so empty text and missing values cannot be told apart when read back. — *to xml (10 paths)*318- Binary values are written as Base64 text. — *to yaml (10 paths)*319- TOML has no null: empty values are omitted (or stop the conversion, as chosen). — *to toml (10 paths)*320- Property lists have no null (empty values are omitted or stop the conversion) and are built in memory (20 million cells at most). — *to plist (10 paths)*321- Column types are detected from the text (verified on every row); numbers are re-written in canonical form (1.50 → 1.5). Identifiers with leading zeros stay text. Turn detection off to keep every cell as written. — *from csv, tsv, xml (40 paths)*322- Same-format conversion: the file is re-written with the chosen options. — *csv→csv, tsv→tsv, jsonl→jsonl, parquet→parquet*323- Integers beyond 64 bits are read as 64-bit floats by the JSON reader. Text that looks like an ISO date becomes a date. — *from json, jsonl (25 paths)*324- The table is built from the document's list of records (detected, or given by “Record list”); other content of the document is not converted. — *from json, plist, toml, xml, yaml (50 paths)*325- XML carries text only: numbers and booleans become text, empty strings and nulls both become empty elements. Keys starting with “@” become attributes, “#text” becomes element text, lists become repeated elements. — *json→xml, xml→xml, yaml→xml, toml→xml, plist→xml*326- TOML has no null (nulls are omitted or stop the conversion, as chosen) and needs a table at the top level; plain values are moved before sub-tables. — *json→toml, xml→toml, yaml→toml, toml→toml, plist→toml*327- Property lists have no null and no key order (keys are sorted when read back); integers beyond 64 bits and exact decimals are written as text. — *json→plist, xml→plist, yaml→plist, toml→plist, plist→plist*328- ZSTD-compressed Arrow buffers, union, run-end-encoded and view types are not supported. Nested columns are read as JSON text; unsigned 64-bit integers become exact decimals. — *from arrow (14 paths)*329- Only tables are read. Views, indexes, constraints and triggers are not carried over; stored SQL is never executed. — *from duckdb, sqlite (28 paths)*330- One table per output file: the first one by default, a chosen one, or all of them as separate files. — *from duckdb, sqlite (20 paths)*331- Dates and binary values become text (ISO 8601 / Base64); JSON has no such types. — *xml→json, yaml→json, toml→json, plist→json*332- XML comments and processing instructions are dropped; attribute order is not kept; every value is read as text. Documents declaring entities (DTD) are refused. — *xml→json, xml→xml, xml→yaml, xml→toml, xml→plist*333- The document is re-serialised: comments and original formatting are not kept. — *xml→xml, yaml→yaml, toml→toml*334- Comments, anchors and formatting of the source are not kept. — *xml→yaml, yaml→yaml, toml→yaml, plist→yaml*335- YAML comments, anchors and tags are not carried over. — *yaml→json, yaml→xml, yaml→toml, yaml→plist*336- TOML comments are not carried over. — *toml→json, toml→xml, toml→yaml, toml→toml, toml→plist*337338339## 3. Type mapping340341Internal column kinds: `string, integer (64-bit), decimal (exact digits), double, boolean, date, timestamp (no zone),342timestampTZ (UTC instant), time, binary, json (nested value)`. Dates/times are integers (days, microseconds):343no calendar or time-zone arithmetic is applied, so nothing is shifted.344345### Reading346347| Source | integer | decimal | double | boolean | date / timestamp | text | other |348|---|---|---|---|---|---|---|---|349| CSV / TSV / XML records | text `-?(0\|[1-9][0-9]*)` in 64-bit range → BIGINT; wider → DECIMAL(38,0) | > 15 significant digits → DECIMAL(38,s) | other numbers (optional decimal comma / thousands separator) | `true/false` (any case) | ISO 8601 date, date-time (→ TIMESTAMP), date-time with offset (→ TIMESTAMPTZ, instant kept), `HH:MM:SS` → TIME; `d/m/y` or `m/d/y` only when unambiguous or when a format is given | everything else, incl. leading-zero identifiers, `+33`, mixed columns | detection is verified on **every** row; one non-conforming value keeps the column text |350| JSON / JSON Lines | BIGINT (UBIGINT/HUGEINT when needed) | — | DOUBLE | BOOLEAN | ISO date/time strings → DATE/TIMESTAMP/TIMESTAMPTZ/TIME (verified on every row) | VARCHAR | objects → STRUCT, arrays → LIST (kept, flattened or JSON text, see §5); integers beyond 64 bits → DOUBLE (DuckDB reader) |351| YAML / TOML / plist / JSON object | as JSON; integers beyond 64 bits and long decimals are carried as **text** (exact) | | | | native dates of the format | | plist `Data`, YAML `!!binary` → Base64 text |352| Parquet / DuckDB | native DuckDB types, unchanged | | | | | | nested types kept / flattened / JSON text |353| XLSX / XLS | number cell whose text is plain digits | — | other number cells | `t="b"` | number + date/time number format (built-in ids and custom codes), 1900 and 1904 systems; `t="d"` ISO cells | shared, inline, formula-string cells | errors → NULL (warned); mixed column → text |354| ODS | `float/percentage/currency` with integer text | — | other floats | `boolean` | `date` (date or date-time), `time` (duration < 24 h) | `string` | mixed column → text |355| SQLite | INTEGER storage | — | REAL | INTEGER 0/1 in a column declared `BOOL*` | TEXT ISO values in columns declared `DATE*`/`TIME*` | TEXT | BLOB → binary; a column mixing storage classes → text |356| Arrow | int8…int64, uint8…uint32 → BIGINT; uint64 → UBIGINT | decimal128 → DECIMAL(p,s) | float16/32/64 | bool | date32/64, time32/64, timestamp (s/ms/µs/ns; ns truncated to µs with a warning; zoned → TIMESTAMPTZ) | utf8, large_utf8, dictionary-encoded strings | binary, large/fixed-size binary; list/struct/map/fixed-size-list → JSON text; duration → BIGINT |357358### Writing359360| Target | integer | decimal / 128-bit | double | boolean | date | timestamp | timestampTZ | time | binary | nested | NULL vs "" |361|---|---|---|---|---|---|---|---|---|---|---|---|362| CSV / TSV | digits | exact digits | shortest round-trip | `true/false` | ISO (or `strftime` option) | `YYYY-MM-DDTHH:MM:SS[.ffffff]` | same + `Z` | ISO | Base64 | JSON text | NULL = empty field (or chosen text), "" = `""` |363| JSON / JSON Lines | number | number, digit for digit | number; NaN/±inf → `null` (counted warning) | `true/false` | ISO string | ISO string | ISO string + `Z` | string | Base64 string | real JSON | `null` vs `""` |364| Parquet | INT64 etc. (native) | DECIMAL; 128-bit ints → DECIMAL(38,0), or text if 39 digits | DOUBLE/FLOAT | BOOLEAN | DATE | TIMESTAMP (µs) | TIMESTAMP (UTC) | TIME | BYTE_ARRAY | native STRUCT/LIST/MAP | native |365| DuckDB | native | native | native | native | native | native | native | native | BLOB | native | native |366| Arrow | int64 | decimal128(p,s); 128-bit ints as Parquet | float64 | bool | date32 | timestamp[us] | timestamp[us, UTC] | time64[us] | binary | utf8 (JSON text) | validity bitmap |367| SQLite | INTEGER | TEXT (exact) | REAL (NaN → text `nan`) | `BOOLEAN` 0/1 | `DATE` ISO text | `DATETIME` ISO text | `DATETIME` ISO text + `Z` | `TIME` text | BLOB | TEXT (JSON) | NULL vs '' |368| XLSX / XLS | number; beyond ±2^53 → text (option: number) | ≤ 15 digits → number, else text | number; NaN/±inf → empty (counted) | boolean | date cell (`yyyy-mm-dd`), outside 1900-03-01…9999 → ISO text | date-time cell, **millisecond** precision (finer → rounded, counted) | UTC wall clock, zone dropped (warned) | time cell | Base64 text | JSON text | empty cell vs empty string cell (XLS: both empty) |369| ODS | `float` | as XLSX | `float` | `boolean` | `date` | `date` with full microseconds | UTC wall clock (warned) | `time` duration | Base64 text | JSON text | empty cell vs empty string cell |370| XML (records) | text | text | text | text | text | text | text | text | Base64 | JSON text → text | NULL = element omitted; "" reads back as NULL |371| YAML (list of mappings) | int | float literal, exact digits | float (`.nan`, `.inf`) | bool | timestamp | timestamp | timestamp `Z` | quoted string | Base64 string | real nested YAML | `null` vs `""` |372| TOML (`[[rows]]`) | integer | float literal / string for > 64-bit integers | float (`nan`, `inf`) | boolean | local date | local date-time | offset date-time | local time | Base64 string | inline tables / arrays | NULL omitted (or error) |373| plist (array of dicts) | integer | string (exact) | real | boolean | string | string | `Date` (XML plists: to the second, warned) | string | `Data` | nested dict/array | NULL omitted (or error) |374375## 4. Reading and writing options (all exposed as `OptionDescriptor`s)376377Reading: `encoding.input`, `csv.delimiter.input`, `csv.quote.input`, `csv.lenient`, `nulls.input`, `header`378(auto/yes/no), `header.row`, `types.detect` (off = all text), `types.columns` (`name=text|integer|decimal|boolean|379date|timestamp`), `numbers.decimal.comma`, `numbers.thousands`, `dates.input.format`, `sheet` (name or `#n`),380`sheets.all`, `formulas.as.text`, `records.path`, `nested` (auto/flatten/json/native), `columns.select`,381`columns.rename`.382Writing: `csv.preset` (standard / Excel-friendly = UTF-8 BOM + CRLF), `csv.delimiter`, `csv.header`,383`csv.quote.all`, `csv.line.ending`, `encoding.output`, `encoding.bom`, `nulls.output`, `dates.output.format`,384`formula.guard` (CSV-injection guard, **off by default**: prefixes text cells starting with `= + - @` with `'`,385counted and reported), `json.pretty`, `xml.root.name`, `xml.row.name`, `xml.item.name`, `nulls.policy`386(omit/error, TOML and plist), `records.key`, `plist.format`, `parquet.compression`387(zstd/snappy/gzip/lz4/uncompressed), `parquet.row.group.size`, `arrow.format` (file/stream),388`sheet.header.bold`, `sheet.header.freeze`, `sheet.overflow` (error/split), `integers.big`, `xlsx.strings`,389`table.name`, `memory.limit.mb`, `threads`.390391- **Encoding detection**: BOM (UTF-8, UTF-16 LE/BE) → UTF-8 validity → UTF-16 without BOM (NUL pattern) →392 single-byte heuristic (no 0x80–0x9F bytes: ISO-8859-1; bytes undefined in Windows-1252 or accented letters inside393 words: Mac Roman; otherwise Windows-1252). A heuristic answer raises the `encoding-guessed` warning. Non-UTF-8394 input is transcoded to UTF-8 in the workspace before DuckDB reads it.395- **Delimiter**: engine sniffer over comma, semicolon, tab, pipe (quote-aware, first 64 KB); quote/escape/header:396 DuckDB's sniffer for that delimiter. Rows of unequal length stop the conversion with an explanation unless397 `csv.lenient` is on.398- **NULL on input**: an unquoted empty field is NULL, `""` is an empty string; extra NULL tokens via `nulls.input`399 apply to unquoted fields only.400- **Sheets/tables**: to a workbook or database target, every sheet/table is carried by default; to a single-table401 target, the first visible one (others listed by name in a `sheets-dropped` data-loss warning) or all of them as402 `<name>-<sheet>.<ext>` files.403404## 5. Nested values (JSON, Parquet, DuckDB, Arrow, YAML/TOML/plist/XML records)405406| `nested` | effect |407|---|---|408| `flatten` (default for flat targets) | each STRUCT field becomes a column `parent.child` (recursively); lists and maps become JSON text |409| `json` | every nested column becomes one JSON-text column |410| `native` (default for Parquet, DuckDB, JSON, JSON Lines targets) | nested types are kept as they are |411412Arrow nested columns are always read as JSON text. A JSON-text column written to JSON/YAML/TOML/plist is re-expanded413into real nested values.414415## 6. No silent loss — warnings416417`preflight` opens the real file exactly as `convert` would (first 20 000 rows per table; counts taken from a partial418read say so) and adds the target's schema-level effects; `convert` raises the same warnings with exact counts.419Codes: `sheets-dropped` (names), `formulas-dropped` / `formulas-as-text` (count), `error-cells`, `merged-cells`,420`hidden-rows-columns`, `hidden-sheet`, `sheet-features-dropped` (data validation, conditional formatting,421hyperlinks), `workbook-content-dropped` (charts, images, pivot tables, comments, external links, macros, defined422names — detected from the package parts), `blank-rows-skipped`, `mixed-types`, `columns-renamed`,423`columns-not-selected`, `encoding-guessed`, `undecodable-characters`, `unencodable-characters`,424`ambiguous-date-format`, `type-detection-reverted`, `timezone-normalized`, `timezone-dropped`,425`sub-millisecond-dropped`, `sub-microsecond-dropped`, `plist-subsecond-dropped`, `integer-as-text`,426`decimal-as-text`, `int128-as-decimal`, `int128-as-text`, `wide-number-as-text`, `non-finite-dropped`,427`date-as-text`, `time-as-text`, `binary-as-text`, `cell-too-long`, `csv-injection-guarded`, `nested-flattened`,428`nested-stringified`, `json-number-precision`, `spreadsheet-number-precision`, `spreadsheet-time-precision`,429`content-outside-records`, `records-located`, `single-record`, `duplicate-keys`, `non-string-keys`,430`yaml-multiple-documents`, `xml-comments-dropped`, `xml-processing-instructions-dropped`, `xml-mixed-content`,431`xml-sibling-order`, `xml-names-adapted`, `xml-empty-strings`, `xml-types-as-text`, `toml-root-wrapped`,432`key-order-changed`, `null-omitted`, `control-characters-replaced`, `views-not-converted`, `sheets-renamed`,433`tables-renamed`, `sheet-split`, `xls-via-libreoffice`, `xls-column-limit`.434435`validate` re-opens the output in a fresh locked-down session and checks: it opens; number of tables; **exact row436count** read back vs rows written; column count; column names; and the cells of the first 1 000 rows of each table437against an independent read of the source (numbers compared by value, date-times by instant). On `lossless` paths a438single differing cell fails validation; on the others the comparison is reported (test `validationCatchesAWrongOutput`).439Tree paths: the output is parsed back; `lossless` paths must be identical, others report the comparison of all440scalar values.441442`inspect` returns: sheets/tables with rows × columns (workbooks are scanned up to 96 MB of compressed sheet data;443beyond that counts are "at least"), formulas, merged ranges, hidden rows/columns, charts/images/pivots/macros;444text encoding, delimiter, quote, header, detected column types; Parquet created_by, row groups, format version,445compression codecs and physical/logical schema; SQLite page size, page count, user_version, application_id,446encoding, journal mode, tables/views with row counts and declared column types; DuckDB table list with row counts447and schema (and the DuckDB version used to read it); Arrow variant, row count and Arrow types; for documents, the448top-level shape, the detected record list and the reader's notes.449450## 7. Security limits451452| Limit | Value | Behaviour |453|---|---|---|454| ZIP entries | 200 000 | `limitExceeded` |455| Small package part (workbook, styles, rels) | 256 MB decompressed | `limitExceeded` |456| Streamed part (worksheet, content.xml, shared strings) | 16 GB decompressed, counted on bytes actually produced, never on declared sizes; CRC-32 verified | `limitExceeded` / `malformedInput` |457| Shared strings in memory | 1 GB | `limitExceeded` |458| Sheet | 1 048 576 rows × 16 384 columns | `limitExceeded` |459| ODS repeated rows/cells | materialised cells ≤ 500 million; trailing empty repeats cost nothing; a repeat that cannot fit is refused before any row is produced | `limitExceeded` |460| XML token (package parts) | 128 MB | `limitExceeded` |461| DOCTYPE / entities | refused everywhere: package parts (tokenizer has no DTD support) and XML documents (byte pre-check + parser delegate; `shouldResolveExternalEntities = false`) | `malformedInput` |462| Tree documents | 512 MB file, 256 levels, 50 million values; YAML alias expansion bounded by input size | `limitExceeded` |463| SQLite | read-only, defensive mode, `trusted_schema` off, views/triggers disabled, no ATTACH, value ≤ 512 MB; only tables are read | error |464| Arrow | message header ≤ 256 MB, decompressed buffer ≤ 2 GB, every offset/length checked | error |465| Encrypted input | OLE-wrapped (password-protected) XLSX, encrypted ZIP entries, encrypted ODS | `encryptedInput` |466| Macros | never executed (VBA parts are only counted) | — |467468## 8. Performance (measured)469470Release build (`swift test -c release -Xswiftc -enable-testing --filter Benchmarks`, harness471`Tests/EngineDataTests/Benchmarks.swift`), one process per benchmark so that the peak resident size (`ru_maxrss`) is472that benchmark's. Inputs generated by DuckDB: 9 columns (integer id, 32-char token, text, decimal amount, boolean,473date, timestamp, category, zero-padded zip). Times include type detection + its full-file verification pass and474exclude `validate`. Defaults: memory limit 4 096 MB, 8 threads, unless stated.475476**The Mac was shared with other heavy builds and test runs during these measurements (1-minute load average between47723 and 84 on 18 cores).** The numbers are what was observed, not best cases; they vary from run to run.478479| Conversion | Rows | Source | Output | Time | Rows/s | Source MB/s | Peak RSS | `validate` | Load avg (1 min) |480|---|---|---|---|---|---|---|---|---|---|481| CSV (2.36 GB) → Parquet, zstd | 22 000 000 | 2 363 MB | 538 MB | 6.90 s | 3 187 274 | 342 | 1 558 MB | 0.29 s, all passed | 83.84 |482| CSV (2.36 GB) → Parquet, snappy | 22 000 000 | 2 363 MB | 1 226 MB | 4.57 s | 4 818 602 | 518 | 1 403 MB | 0.30 s, all passed | 45.57 |483| CSV (2.36 GB) → Parquet, zstd, memory limit 1 024 MB | 22 000 000 | 2 363 MB | 538 MB | 8.25 s | 2 666 335 | 286 | 1 210 MB | 0.24 s, all passed | 44.72 |484| CSV (2.36 GB) → Parquet, zstd, memory limit 512 MB, 4 threads | 22 000 000 | 2 363 MB | 538 MB | 7.49 s | 2 936 470 | 315 | 812 MB | 0.16 s, all passed | 44.74 |485| CSV (2.36 GB) → Parquet, zstd, type detection off | 22 000 000 | 2 363 MB | 462 MB | 2.43 s | 9 067 708 | 974 | 1 961 MB | 0.17 s, all passed | 40.77 |486| Parquet (zstd) → CSV | 22 000 000 | 538 MB | 2 363 MB | 3.10 s | 7 096 124 | 174 | 2 071 MB | 1.13 s, all passed | 40.77 |487| Parquet (zstd) → CSV, memory limit 512 MB, 4 threads | 22 000 000 | 538 MB | 2 363 MB | 4.17 s | 5 276 365 | 129 | 758 MB | 1.35 s, all passed | 46.02 |488| CSV → XLSX (1 M rows, shared strings) | 1 000 000 | 105 MB | 73 MB | 6.16 s | 162 428 | 17 | 384 MB | 4.66 s, all passed | 45.38 |489| CSV → XLSX (1 M rows, inline strings) | 1 000 000 | 105 MB | 79 MB | 4.44 s | 225 321 | 24 | 207 MB | 3.87 s, all passed | 45.43 |490| XLSX (1 M rows) → CSV | 1 000 000 | 73 MB | 105 MB | 4.55 s | 219 933 | 16 | 552 MB | 1.00 s, all passed | 39.44 |491| CSV → ODS (1 M rows) | 1 000 000 | 105 MB | 73 MB | 5.43 s | 184 246 | 20 | 206 MB | 8.83 s, all passed | 37.65 |492| JSON Lines (1.45 GB) → Parquet, zstd | 8 000 000 | 1 480 MB | 195 MB | 12.99 s | 615 644 | 114 | 3 685 MB | 0.46 s, all passed | 31.53 |493| CSV (2.36 GB) → SQLite | 22 000 000 | 2 363 MB | 2 431 MB | 33.07 s | 665 267 | 71 | 503 MB | 0.66 s, all passed | 26.86 |494| CSV (2.36 GB) → Arrow IPC file | 22 000 000 | 2 363 MB | 2 094 MB | 13.43 s | 1 637 648 | 176 | 443 MB | 0.20 s, all passed | 22.64 |495496497Reading the table: CSV → Parquet reads the 2.36 GB file twice (verification of detected types, then the copy); with498detection off it is read once. With a 512 MB limit the same 2.36 GB conversions complete in well under the file's499size in resident memory, which is the larger-than-memory proof asked for. JSON Lines → Parquet detects the schema on500the whole file before converting; its peak of 3.7 GB was measured with the default 4 GB limit and was not measured501with a lower one. CSV → SQLite is bound by single-threaded row inserts. XLSX → CSV keeps the workbook's shared502strings in memory (1 M distinct 32-character tokens here).503504## 9. Evaluated and NOT implemented505506- **Apache `arrow-swift`** (evaluated at commit `15dfba80`, 2026-11-10): it builds with Swift 6.4.2 (tools 5.10,507 Swift 5 language mode) but pulls 21 transitive packages (grpc-swift, swift-nio, swift-protobuf… for Arrow Flight),508 and its reader failed on files written by Arrow C++ (pyarrow 24): `Unsupported field type found: largeutf8`, and509 `…: decimal` on a file with a decimal128 column. Rejected; the engine has its own reader/writer, cross-checked510 during development against pyarrow 24 in both directions (pyarrow `validate(full=True)` on our files); the tests511 use four small files written by pyarrow (`Tests/EngineDataTests/Fixtures`) and a byte-level layout test, and do512 not need Python.513- **Arrow**: ZSTD-compressed buffers (no public Zstandard decoder on macOS; clear error telling to re-save without514 ZSTD), union, run-end-encoded, string-view/binary-view, list-view, interval, decimal256, big-endian files,515 dictionary-encoded or nested columns on **write**, buffer compression on write, Feather v1.516- **DuckDB extensions** (sqlite, excel, arrow, spatial, httpfs…): not used by design (no download, no network).517- **Native BIFF8 (XLS) parser**: not written; XLS goes through the bundled LibreOffice (a `soffice` process is started for each518 conversion; its cost was not benchmarked). Without LibreOffice in the build, XLS paths are not declared.519- **XLSX writer**: formulas, styles beyond the bold frozen header and date formats, column widths, charts, multiple520 header rows, ZIP64 (a package part over 4 GB stops with `limitExceeded`). **XLSX reader**: formulas are not521 evaluated (cached values are read); rich-text formatting, comments and hyperlinks are dropped (reported); XLSB and522 password-protected workbooks are not read.523- **ODS**: freeze panes on write; formula evaluation; sub-tables.524- **Views** in SQLite and DuckDB files: listed, never executed. Indexes, constraints and triggers are not carried525 between databases. SQLite → SQLite and DuckDB → DuckDB are not offered (copy the file).526- **XML**: DTDs/entities are refused (also harmless ones); namespaces are kept as written prefixes, not resolved;527 mixed content keeps its text but not its position; documents over 512 MB (tree in memory).528- **JSON tabular input**: integers beyond 64 bits become doubles (DuckDB reader) for top-level arrays and JSON529 Lines; exact handling only for documents read by the engine's parser (JSON objects, YAML, TOML, plist, where such530 numbers are carried as text).531- **YAML**: comments, anchors, custom tags and block styles are not preserved; `!!binary` is written as a Base64532 string (Yams' emitter does not write tags). **TOML**: comments not preserved. **plist**: key order (Apple's API).533- **Text encodings** other than UTF-8, UTF-16 LE/BE, Windows-1252, ISO-8859-1, Mac Roman.534- **Geospatial and scientific formats** (GeoJSON, GPKG, HDF5, NetCDF…): out of scope of this engine by assignment.535536## 9b. XML mapping of a document's top level (fixed 2026-11-21)537538An XML document has exactly one root element. JSON, YAML and property lists may hold anything at their top level, so539the tree → XML writer maps the top level as follows (`Tree/XMLTree.swift`, `XMLTreeCodec.Writer.serialize`):540541| Top-level value | Written as | Reads back (XML → JSON) as |542|---|---|---|543| object with one key whose value is not a list | that key is the root element | the same object |544| any other object | `<root>` holding the members | `{"root": {…}}` |545| list `[a, b, …]` | `<root><item>a</item><item>b</item>…</root>` | **the same list** (see the rule below) |546| list inside a list | `<item>` holding `<item>` elements | `{"item": [...]}` per inner list (XML has no anonymous list) |547| empty list | `<root/>` (warning `xml-empty-list`) | `null` |548| scalar / text | `<root>text</root>` | `{"root": "text"}` |549| null | `<root/>` | `{"root": null}` |550551Reading rule that makes top-level lists round-trip: a root element named `root` whose only children are `item`552elements (no attribute, no text) is read as a list of those items. Consequences, all intended: `[{"id":1},{"id":2}]`553→ XML → JSON gives a list of two records again (values as text, as for every XML path); a one-entry list stays a554list; an object written by hand as `{"item": [...]}` also reads back as a list. The element names can be changed555(`xml.root.name`, `xml.item.name`); with other names the list is found under `<root name>.<item name>` when read back,556and the `xml-root-wrapped` warning says so. An empty list under a key is written as one empty element and reported557(`xml-empty-list`) instead of vanishing.558559Before this fix a top-level list of two or more entries was written as several root elements (`<root>…</root>560<root>…</root>`): not XML. The engine's own validation caught it, so `json→xml`, `yaml→xml` and `plist→xml` failed561on every such document. Tests: `TreeTests.topLevelValuesToXML` (27 cases: nine shapes × JSON, YAML, plist),562`TreeTests.topLevelListRoundTripsThroughXML`.563564## 10. Known defects and caveats565566- The official DuckDB library is a 62 MB dynamic library declaring macOS 11.0 as its minimum; it must be embedded and567 signed with the app (`Contents/Frameworks`).568- An unterminated quote in a CSV is accepted by DuckDB (the field runs to the end of the file) rather than rejected.569- A zero-byte file with a SQLite extension is a valid empty database for SQLite: it inspects fine and `convert`570 answers "contains no ordinary table".571- XLSX output with shared strings keeps up to 1 000 000 distinct strings / 256 MB in memory, then switches to inline572 strings; reading a workbook loads its shared-string table in memory (1 GB cap).573- `validate` reads the output again: for large XLSX/ODS outputs this costs about as much as the conversion.574- Preflight on very large documents (> 64 MB tree formats) is skipped and says so.575- Tree → XML: a list nested directly in a list, an empty list and a top-level scalar do not read back as what they576 were (see §9b); an object whose only key is `item` under a root named `root` reads back as a list.577- `DataTests.cancellationStopsMidStream` (CSV → Parquet case) is timing-sensitive: on a heavily loaded Mac (load578 average above 40 on 18 cores) the conversion once finished before the cancellation was seen and the test failed;579 it passed when re-run alone. The test, not the engine, depends on the load.580581## 11. Tests582583`swift test` from `Packages/EngineData` (debug), 2026-11-21, with the bundled LibreOffice present:584`Test run with 70 tests in 6 suites passed after 55.780 seconds` — 67 tests run (297 parameterised cases, among them585the 194 tabular and 25 tree paths and the 27 top-level XML shapes of §9b), 3 benchmark entries disabled unless586`ORIX_DATA_BENCH` is set.587588| Suite | What it covers |589|---|---|590| `MatrixTests` | every declared tabular path: convert the canonical dataset (unicode incl. emoji and RTL, quotes, delimiters, line breaks, integers beyond 2^53, decimals, booleans, dates, timestamps, NULL vs empty string, leading-zero identifiers), run `validate`, read back and compare every cell |591| `TreeTests` | every declared tree path; JSON exactness (key order, big integers, decimal text, escapes, surrogates), YAML typing and quoting, TOML 1.0 features and rejections, XML mapping convention and its reversibility, XXE (file:// entity, parameter entity, UTF-16), billion laughs, YAML alias bomb, 100 000-level nesting, null policies, malformed documents |592| `SpreadsheetTests` | hand-assembled Excel-style packages (shared/inline/rich strings, phonetic runs, `_xHHHH_` escapes, built-in and custom date formats, 1900 and 1904 systems, cached formula values, shared formulas, errors, merged cells, hidden sheets/rows/columns, prefixed elements, charts/images/pivots/comments/macros parts), number-format classification, writers' edge cases and loss tallies, sheet overflow (error and split), ODS typed cells and repeats, repeat bombs, ZIP reader (bomb limit, truncation, checksum, `unzip -t`), tokenizer, LibreOffice cross-check of our XLSX and ODS (conditional), XLS limits (conditional) |593| `TextTests` | delimiter/quote/header sniffing (semicolon + decimal comma, tab, pipe, quoted line breaks), encodings (UTF-8 ± BOM, UTF-16 LE/BE ± BOM, Latin-1, Windows-1252, Mac Roman, forced), conservative type detection and its full-file verification, forced types, NULL tokens, skipped rows, ragged rows, writing options, Excel preset, CSV-injection guard, column selection/renaming |594| `DataTests` | declared contract (path count, options, defaults), Arrow files written by Arrow C++ (plain, LZ4, stream), ZSTD refusal, damaged Arrow, byte-level IPC layout of our files, SQLite (types, views, triggers, hostile names, read-only hardening), DuckDB files (all types through every streamed target), Parquet codecs/row groups/metadata and broken footers, nested JSON (flatten/json/native, record paths), damaged inputs for every reader, hostile file/table/column names, the DuckDB sandbox, memory limit and spilling, cancellation mid-stream for five targets, preview API, validation catching wrong outputs |595