SPB Git

Repository Public

NATIVE-FORMAT.md 10.9 KB

# The .orixsheet document format — version 1

.orixsheet is the native, lossless format of ORIX Sheet. A document is a package: a directory that macOS presents as a single file. The application reads and writes it by URL (NSDocument.read(from:ofType:) / write(to:ofType:) calling WorkbookReader.load and WorkbookWriter.save); a FileWrapper API exists as well and is tested, but the app does not use it. Everything the SheetCore model holds is stored; nothing is recomputed on load.

Implementation: Sources/SheetIO/Native/ (NativeFormat, NativeSchema, NativeCells).

text
Budget.orixsheet/
├── manifest.json            format id, versions, dates, SHA-256 and size of every other part
├── workbook.json            sheet list, defined names, style table, properties, attachment index
├── sheets/
│   ├── 1.cells              cells of the sheet whose id is 1 (binary, chunked, zlib)
│   ├── 1.meta.json          everything in SheetMeta for that sheet
│   ├── 4.cells
│   └── 4.meta.json
├── attachments/
│   └── 0-clean.py           Workbook.attachments (Python scripts, workflows, queries…)
├── preserved.plist          optional: parts kept from an imported XLSX (charts, VBA, pivots…)
└── preview.png              optional: thumbnail passed in `NativeWriteOptions.preview` (the app passes none)

Sheet parts are named after Sheet.id, which is stable across renames and reordering. The order of the sheets is the order of workbook.json › sheets.

# API

swift
NativeFormat.write(workbook, to: url, options:, preserved:)      // atomic, returns [CompatibilityIssue]
NativeFormat.read(url:, options:, progress:) -> LoadResult
NativeFormat.fileWrappers(for: workbook, options:, preserved:) -> FileWrapper   // not used by the app
NativeFormat.read(fileWrapper:, options:, progress:) -> LoadResult             // not used by the app
NativeFormat.verify(url:) -> [String]                           // paths failing the integrity check

WorkbookReader.load / WorkbookWriter.save with .orixsheet call the same code; this is the path the application takes (App/Sources/WorkbookDocument.swift). The two FileWrapper functions build and read the same parts in memory for a host that prefers NSDocument.fileWrapper(ofType:); they are covered by NativeFormatTests.fileWrappersPreviewAttachmentsAndPreservedParts only.

preview.png is written when the caller sets NativeWriteOptions.preview. The application does not set it, so documents saved by ORIX Sheet contain no preview; a reader must not expect one.

# manifest.json

json
{
  "format": "orixsheet",
  "formatVersion": 1,
  "minReaderVersion": 1,
  "app": "ORIX Sheet",
  "appVersion": "1.0",
  "created": 721692800.0,
  "modified": 785000000.5,
  "parts": {
    "workbook.json":      { "sha256": "9f2c…", "size": 1830 },
    "sheets/1.cells":     { "sha256": "41aa…", "size": 5242880 },
    "sheets/1.meta.json": { "sha256": "c07e…", "size": 912 }
  }
}
  • parts lists every file of the package except the manifest itself, with its SHA-256 (lower-case hex) and byte size.
  • Dates everywhere in the JSON parts are seconds since 2001-01-01 00:00:00 UTC as a double (Date.timeIntervalSinceReferenceDate). This is the only encoding that round-trips a Swift Date bit for bit; ISO 8601 would lose sub-millisecond precision.
  • created is Workbook.properties.created when set, otherwise the time of the first save.

# workbook.json

Key Content
sheets array, in tab order: id, name, cells (part path), meta (part path), cellCount
names defined names: key (dictionary key in Workbook.names), name, formula, scopeSheetID, comment, hidden
styles the StyleTable in index order; cells refer to styles by index, index 0 is the default style
date1904, activeSheet, defaultFontName, defaultFontSize workbook settings
properties WorkbookProperties (title, author, subject, keywords, comments, created, modified)
attachments name (key in Workbook.attachments) → path inside the package
hasPreservedParts present and true when preserved.plist exists

Attachment names are arbitrary strings; the file name inside attachments/ is derived from the index and a sanitised suffix ([A-Za-z0-9._-]), so a name can never escape the package.

# sheets/<id>.meta.json

A direct image of SheetMeta: colWidths, rowHeights (points, keyed by index), default sizes, hiddenRows / hiddenCols / filteredRows / collapsed sets (index ranges), outline levels, row/column styles, merges, frozen panes, tables, conditionalFormats, validations, comments and hyperlinks (arrays of {row, col, value}), autoFilter, objects (SheetObject, payload as base64), print, view flags, tabColor, isHidden, isProtected.

Number values are written with the shortest representation that reads back to the same double. Non-finite numbers are written as the strings "+inf", "-inf", "nan".

# sheets/<id>.cells

Binary, little-endian. Designed for millions of cells: column-major order (the order of the in-memory store), independent zlib-compressed chunks of 50,000 cells, per-chunk string table.

text
header   4 bytes  "OXCL"
         u32      version (1)
         u64      number of cells
         u32      number of chunks
chunk    u32      raw (uncompressed) length
         u32      compressed length
         u32      number of cells in the chunk
         u32      CRC-32 of the raw bytes
         bytes    zlib stream (RFC 1950)

Raw chunk content (varint = LEB128 unsigned, zigzag = zigzag-encoded signed varint):

text
varint stringCount, then stringCount × (varint byteLength, UTF-8 bytes)
then, for every cell:
  varint  column delta from the previous cell (the first cell of a chunk: absolute column)
  varint  row: delta from the previous cell when the column is unchanged, absolute otherwise
  u8      flags: bits 0–3 value kind, 0x10 has formula, 0x20 isSpill, 0x40 has style
  zigzag  style index                              (when 0x40)
  …       value payload (see below)
  when 0x10:
    u8      formula flags: 1 = spill range present, 2 = volatile
    varint  string index of the formula text (canonical text, no leading "=")
    u8      kind of the cached value, then its payload
    4 × varint  r0, c0, r1, c1 of the spill range  (when flag 1)

Value kinds and payloads:

Kind Meaning Payload
0 blank –
1 number 8 bytes, IEEE-754 double (keeps −0.0, subnormals, NaN payloads)
2 integral number zigzag (exact integers below 9·10¹⁵)
3 text varint string index
4 / 5 FALSE / TRUE –
6 error u8: Excel ERROR.TYPE code (1 #NULL!, 2 #DIV/0!, 3 #VALUE!, 4 #REF!, 5 #NAME?, 6 #NUM!, 7 #N/A, 9 #SPILL!, 14 #CALC!, 15 #CIRC!)
7 array varint rows, varint cols, then rows × cols × (u8 kind, payload)
8 LAMBDA value – (cannot be stored; read back as #CALC!, reported as a loss on save)

Formulas are stored as canonical formula text (Expr.formulaText(), the OOXML en-US syntax) and parsed again on load; identical texts inside a chunk are parsed once. The cached value, the spill range and the volatile flag are stored next to it so a document opens without recalculation (CalcEngine(workbook:trustCachedValues: true)).

# preserved.plist

Binary property list of PreservedParts: raw bytes of the XLSX parts ORIX Sheet does not model (charts, drawings, images, pivot tables and caches, VBA project, external links, custom XML, theme) with the sheet ids they belong to. It lets an imported Excel workbook be saved natively and exported to XLSX later without losing those parts. Its content is never interpreted or executed.

# Integrity and recovery

On load every part listed in the manifest is hashed and compared.

Situation Default behaviour LoadOptions.strictIntegrity
workbook.json missing or undecodable SheetIOError.corrupted(part: "workbook.json", …) same
workbook.json hash mismatch but decodable loads, warning error
a .cells part: bad chunk (CRC, zlib, structure) that chunk is skipped, the other chunks and sheets load; loss issue naming the sheet, the part and the chunk corrupted(part: "sheets/4.cells", …)
a .cells part missing or truncated the sheet opens with what could be read (possibly empty); loss issue error
a .meta.json part missing or undecodable cells load, layout of that sheet is reset to defaults; loss issue error
attachment missing / changed loss / warning issue error
manifest missing or unreadable document opens without verification; warning issue error

Issues are returned in LoadResult.issues with location set to the part path. The reader never trusts sizes found in the file: chunk lengths are bounded (256 MiB raw), decompression stops at the declared size, every index is range-checked, and LoadOptions.maxCells caps the total.

# Atomic save

NativeFormat.write builds the complete package in a hidden sibling directory (.<name>.saving-<uuid>), then swaps it in with FileManager.replaceItemAt (or a rename when the destination does not exist). If anything fails before the swap — disk full, crash, power cut — the previous document is untouched and the temporary directory is removed. A successful save replaces the whole package, so parts of deleted sheets never linger.

# Versioning

  • formatVersion is the schema version the writer used; minReaderVersion is the oldest reader that can open the file without misreading it.
  • Reader rules: formatVersion ≤ ours → migrate if older, then read. formatVersion > ours and minReaderVersion ≤ ours → read, ignore what is unknown, warn that unknown content would be lost on save. minReaderVersion > ours → SheetIOError.unsupportedVersion.
  • Forward compatibility: unknown JSON keys are ignored; unknown files in the package are ignored; lists whose elements may gain new variants (tables, conditionalFormats, validations, objects) are decoded element by element, so one unknown element is skipped and reported instead of making the sheet unreadable.
  • Migration hook: NativeFormat.migrations[v] upgrades the parts of a version-v package to v + 1 before decoding; migrate(_:from:) chains the steps. Version 1 is the first released schema, so the table is empty. A future version that changes a part bumps formatVersion, registers a step, and raises minReaderVersion only when old readers would misread the file.
  • The .cells part has its own version field (1) and magic number, checked independently.

# What is not stored

  • LAMBDA values held by a cell (closures): saved as #CALC!, recomputed on load, reported as a loss issue by write and preflight.
  • Engine state that is derived from the model (dependency graph, dirty flags) is rebuilt.
  • The private sheet-id counter of Workbook is restored to "largest id + 1".
Files main
Menu

Theme