SheetData benchmarks
Measured with orix-data-bench (this package). These are measurements of one run on one machine, not guarantees; every number moves with hardware, disk cache state, thread count and data shape.
Setup
| Date | 2026-11-21 |
| Machine | Mac17,6, Apple M5 Max, 18 cores, 48 GiB RAM, macOS 27.0 (26A5425a) |
| Engine | DuckDB v2.0.1 (vendored universal dylib), 18 threads, memory_limit 24 GiB (default = half of RAM), release build |
| Dataset | 10,000,000 synthetic rows × 8 columns: id BIGINT, category VARCHAR (20 values), region VARCHAR (8), order_date DATE (2020–2024), amount DOUBLE, quantity BIGINT (1–50), customer_id BIGINT (100,000 values), status VARCHAR (5) |
| Files | CSV 559.9 MB (written by the tool, streamed through a 4 MiB buffer), Parquet 108.1 MB (zstd, written by DuckDB from the CSV), plus a 100,000-row customers CSV for the join |
| Cache state | the files were written seconds before being read, so they were in the OS page cache: import and scan times are CPU-bound, not disk-bound |
Command:
swift build -c release
.build/release/orix-data-bench --dir /tmp/orix-bench --rows 10000000 > report.jsonQueries are run three times; the table gives the median and the best run. One-off steps (generation, import, profile, writes to the grid) are run once.
Results
| Step | Median (s) | Best (s) | Notes |
|---|---|---|---|
| Generate CSV (Swift, streamed) | 3.931 | 10,000,000 rows, 559.9 MB, process RSS 15 MiB | |
| Register CSV as a lazy view | 0.230 | sniffing only, no rows loaded | |
Aggregate straight from the CSV view (GROUP BY region) |
0.138 | 0.129 | full parse of the file each time |
Convert CSV → Parquet (exportQuery) |
0.671 | 108.1 MB | |
Import CSV into a table (importFile) |
0.300 | 10,000,000 rows, ≈ 33 M rows/s | |
| Row count | 0.0004 | 0.0003 | |
Filtered aggregation (2 IN values, date ≥, amount >, GROUP BY category) |
0.014 | 0.012 | 20 groups |
Pivot region × category by year, sum + % of grand total (PivotEngine.compute) |
0.038 | 0.037 | 169 row items × 6 column items; DuckDB returned 1,014 grouped rows |
Join with customers on customer_id + GROUP BY segment, status (transform pipeline) |
0.088 | 0.065 | 20 groups |
| Preview window offset 5,000,000, limit 200 (insertion order) | 0.005 | 0.003 | |
Preview window sorted by amount DESC, offset 1,000, limit 200 |
0.033 | 0.032 | top-N with a deterministic tie-break |
| Preview window with a filter, offset 10,000, limit 200 | 0.007 | 0.006 | |
| Profile of the 8 columns (+ top 3 values each) | 1.127 | exact quantiles on 10 M rows dominate | |
| Pivot computed, rendered and written to a sheet | 0.146 | 171 × 14 cells, one undo step | |
| Materialize a 100,000-row × 8-column extract into a sheet | 0.925 | 800,008 cells through CalcEngine.set, one undo step |
|
| Materialize the whole table into a sheet | refused | exceedsSheetLimits: 10,000,001 rows × 8 columns do not fit 1,048,576 × 16,384 |
|
Parquet, pushdown filter (id BETWEEN 9,000,000 AND 9,010,000) |
0.004 | 0.004 | row groups skipped from statistics; 10,001 rows matched |
Parquet, projection + aggregate (GROUP BY category, 2 of 8 columns read) |
0.011 | 0.010 | |
| Parquet, the filtered aggregation above | 0.014 | 0.014 | no import needed |
Peak resident memory of the whole process (getrusage, ru_maxrss): 2,941 MiB, including the imported table (in-memory database), the 800,008-cell worksheet and the 1,048,576 rows fetched before the full-table materialization was refused. After the import alone it was 1,628 MiB.
Consistency check at the end of the run: the table, the Parquet view and the CSV view all hold 10,000,000 rows and the same total amount (compared as integer cents).
Two earlier runs of the same binary on the same machine gave 0.297 s and 0.364 s for the import, 0.011 s for the filtered aggregation, 0.029–0.031 s for the pivot, 0.028–0.029 s for the join and 0.003–0.004 s for the middle preview window: expect that kind of spread.
Reading the numbers
- The grid never sees the 10 M rows: the UI reads 200-row windows (
preview), and only bounded results (a pivot, an extract) cross into a worksheet. - Writing to the grid is the slow door: about 0.86 M cells/s through SheetCore's per-cell
set, against tens of millions of rows per second inside the engine. swift testdoes not run this benchmark. A 1,000,000-row end-to-end test exists behindORIX_DATA_LARGE_TESTS=1.