SPB Git

Repository Public

formats.md 24.3 KB

# Format compatibility (DocFormats)

This page covers the formats handled by Packages/ORIXDocsKit/Sources/DocFormats: the native .orixdoc container, Markdown, HTML, RTF, plain text and PDF text import. DOCX is documented with the DocDOCX module.

Nothing here is a benchmark or a conformance score. Every statement below is backed by a test in Tests/DocFormatsTests or is listed as not verified.

# How to read the tables

  • kept — comes back identical after export + import (round trip), or is read faithfully on import;
  • approximated — content kept, form changed; the user gets an approximated warning;
  • dropped — not carried; the user gets a dropped warning (exports) or the content is absent from the source format (imports);
  • DocumentIO.preflight(_:format:) returns the export warnings before exporting, from the document alone. The exporters build their warnings from the same table, plus what they only discover while writing (a missing font, a missing image).

# Summary

.orixdoc Markdown HTML RTF Plain text PDF
Import yes yes yes yes yes text only
Export yes yes yes yes yes no (DocLayout publishes PDF)
Text, Unicode kept kept kept kept up to canonical equivalence ¹ kept as extracted
Paragraph styles kept headings, quote, code, list headings, title, subtitle, abstract, caption, quote, code, list flattened to direct formatting dropped dropped
Bold / italic / strike kept kept kept kept dropped dropped
Underline, sub/superscript, highlight kept inline HTML (<u>, <sub>, <sup>, <mark>), colour of highlight dropped kept kept dropped dropped
Fonts, sizes, colours kept dropped kept (inline styles) kept, fonts substituted when not installed ² dropped dropped
Alignment, indents, spacing kept dropped (table column alignment kept) kept (inline styles) kept dropped dropped
Tabs stops kept dropped dropped (tab characters kept) kept tab characters kept dropped
Links kept kept kept kept text only dropped
Lists, nesting, start number kept kept (9 levels; number formats become digits) kept (start, type) kept labels written as text dropped
Tables kept pipe tables; merged cells become separate cells kept with colspan / rowspan, shading kept with spans and shading; header-row flag dropped tab-separated lines dropped
Images kept kept (data URI or caller-supplied path); display size dropped kept (data URI), size kept PNG / JPEG kept, others converted to PNG; alt text dropped alt text in brackets dropped
Floating placement, crop, rotation kept inline CSS float; crop and rotation dropped inline — —
Shapes kept text only text only text only text only —
Equations kept kept ($…$, $$…$$) kept (data-latex; MathML when a converter is supplied) LaTeX source as text LaTeX source as text —
Footnotes kept kept kept appended at the end, marks as superscript text appended at the end —
Endnotes kept written as footnotes kept appended appended —
Citations kept kept ([@key, p. 4]) or formatted text kept (data-cite-items) + displayed text text text —
Bibliography block kept ::: {#refs} placeholder, or formatted entries kept as a block, entries when a formatter is supplied formatted entries or nothing formatted entries or nothing —
Table of contents, lists of figures kept static list of links kept as a block (<nav class="toc">) static text static text —
Captions, sequence fields kept resolved text resolved text resolved text resolved text —
Cross-references kept resolved text (linked when the target is a heading) resolved text, linked resolved text resolved text —
Page number / page count / running-heading fields kept left empty left empty left empty left empty —
Bookmarks kept kept ({#id} on headings, <a id> elsewhere) kept dropped dropped —
Comments kept dropped dropped dropped dropped —
Tracked changes kept resolved (accepted, or rejected with acceptChanges = false) resolved resolved resolved —
Page breaks kept dropped CSS print break, read back kept form feed kept between pages
Sections, page setup, columns kept dropped dropped paper size and margins only dropped paper size of page 1
Headers and footers kept dropped dropped dropped dropped dropped
Metadata kept title, author, subject, keywords, language (front matter) title, author, description, keywords, language title, author, subject, keywords dropped title, author, subject, keywords
Preserved DOCX parts kept dropped dropped dropped dropped —

¹ The system RTF codec writes precomposed characters where Windows-1252 has them (e + U+0301 comes back as é) and drops the bidirectional embedding controls U+202A…U+202E. The text is canonically equivalent, not always identical. ² Families with no installed named face (the system fonts "New York", "SF Pro Display", "SF Mono") are written as Times New Roman or Helvetica, with a warning.

# Native .orixdoc

Lossless: the round trip of a document using every Block case, every Inline case, assets, notes, comments (with replies), tracked insertions / deletions / format changes, bibliography, sections and preserved parts is compared with ==. Layout, integrity checks, recovery and migrations are described in docs/architecture/persistence.md.

Verified by tests: byte flips at the start, middle and end of each kind of entry (document, manifest, mimetype, asset, preserved part) are detected by read and by validate; a tampered asset inside a well-formed ZIP is caught by the SHA-256; truncation at several lengths; wrong mimetype; mimetype not first; missing manifest; manifest entry missing from the archive; future schema and container versions; unknown extra entries; recover on a damaged asset, a damaged manifest and a truncated file; the migration pipeline with an injected table.

Not verified: files larger than a few megabytes; behaviour at the 1 GiB limit (the limit is a comparison on the byte count, tested as such, not with a gigabyte of data).

# Markdown

Parser and writer are our own (MarkdownImporter, MarkdownInline, MarkdownExporter); the dialect is CommonMark with the GFM and pandoc extensions listed here.

# Import

Construct Result
ATX and setext headings Heading1…Heading6; a pandoc attribute {#id} becomes a bookmark
Paragraphs, soft breaks soft break → space
Hard breaks (two spaces, backslash, <br>) U+2028
*em*, **strong**, _/__ (CommonMark delimiter rules) direct italic / bold
~~strike~~, ~sub~, ^sup^, <u>, <sub>, <sup>, <mark> direct formatting
`code` character style InlineCode
Fenced (```, ~~~) and indented code one Code paragraph per block, lines separated by U+2028; the language label is dropped (the model has nowhere to store it)
Block quotes Quote paragraphs; nested quotes become indented quotes; lists, code, headings and tables inside a quote lose the quote formatting (warning)
Lists ListItem levels 0…8, one ListDefinition per top-level list, start number kept; deeper nesting is flattened to level 8 with a warning; continuation paragraphs are indented paragraphs
GFM task items text prefixed with ☐ / ☑
GFM pipe tables, pandoc simple tables Table, first row is the header, column alignment on the cell paragraphs
---, ***, ___ an empty paragraph with a bottom border (the model has no rule block)
Links, reference links, autolinks, bare http(s):// URLs CharProps.link + Hyperlink style; link titles dropped
Images data: URIs and relative paths through resourceLoader; remote URLs are never fetched: alt text + warning; paths that are absolute, contain .. or a scheme are refused
Footnotes [^id] Note (multi-paragraph definitions supported); unused definitions dropped with a warning; references without definition stay literal
YAML front matter title, author, subject / description, keywords / tags, lang / language; other keys reported
$…$, $$…$$ Equation (inline / display); a $$ block alone in a paragraph is a display-equation paragraph
[@key, p. 4; -@other] Citation with prefix, locator / suffix, suppress-author
::: {#refs} the bibliography block; other fenced divs are transparent
Character references decoded (HTML 4 names and numeric forms)

Not supported on import: pandoc grid and multiline tables, definition lists, inline footnotes ^[…], bare @key citations outside brackets, raw HTML blocks (kept as literal text, with a warning), link titles, image sizes, emoji shortcodes.

# Export

The exporter writes pandoc-flavoured Markdown. Text is escaped so that it reads back as text (including :// and www., so a string that merely looks like a URL does not come back as a link). Emphasis markers are always placed against a word; when CommonMark's flanking rules would otherwise misread a boundary (bold immediately followed by italic, a closing marker after punctuation and before a letter), an empty HTML comment <!----> is inserted. It is invisible when rendered and ignored on import.

Approximations specific to the exporter: the Title paragraph becomes a level 1 heading; heading levels 7–9 become level 6; automatic heading numbers are written as text; consecutive Code paragraphs form one fenced block; a table always gets a header row; empty paragraphs are not written; list number formats (letters, roman numerals) become digits.

A second export of an imported export is byte-identical (tested on the rich fixture).

# Verified with pandoc 3.9.0.2

  • pandoc parses our Markdown export of a document with headings, emphasis, strikethrough, code, link, footnote, inline and display math, three list levels, an ordered list starting at 3, a quote, a code block, a table, a rule, non-Latin text and a hard break: the block sequence of its JSON AST, the inline node types, the link target, the LaTeX of both equations, the code, the note count and the full word sequence match the model. No raw HTML and no citation node appears, i.e. nothing we escaped was read as markup.
  • pandoc -f gfm reads the same export (without math) with the same structure.
  • We import what pandoc writes from one source in four dialects (markdown, gfm, commonmark_x, markdown_strict + extensions) and get the same model as from the source, with these documented differences: GFM has no heading ids (the internal link keeps its target, the heading gets no anchor) and writes math in GitHub's forms, which stay code for us.
  • Our export, rewritten by pandoc and imported again, gives the same structure as importing our export directly.

Skipped when pandoc is not installed.

# HTML

# Import

Our own tokenizer and tree builder (HTMLTokenizer, HTMLImporter). NSAttributedString's HTML import is not used: it runs WebKit and can load remote resources.

Handled: p, div, h1–h6, br, hr, b/strong, i/em/cite/dfn/var, u/ins, s/strike/del, sub, sup, code/kbd/samp/tt, pre, blockquote, a, mark, q, font, center, ul/ol/li (nested, start, type), dl/dt/dd, table/thead/tbody/tfoot/tr/th/td/caption with colspan/rowspan, img, figure/figcaption, title, meta (author, keywords, description), lang, dir, align, bgcolor, character references.

Inline style subset: font-family, font-size, color, background-color, text-align, font-weight, font-style, text-decoration, vertical-align (super / sub), font-variant: small-caps, text-transform: uppercase, white-space: pre*, margin-left / margin-right / margin-top / margin-bottom, text-indent, line-height, direction, display: none, page-break properties.

Recognised conventions: pandoc and ORIX footnotes (a.footnote-ref + section.footnotes), span.math with a data-latex attribute or delimited TeX, MathML with a TeX annotation, span.citation with data-cite-items / data-cites, nav.toc, section.orix-bibliography, .page-break, h1.title, p.subtitle, p.abstract, p.caption.

Security behaviour (all tested):

  • <script> and <style> content is skipped by the tokenizer and never interpreted; scripts are reported as dropped;
  • iframe, frame, object, embed, applet are removed with their content; svg, canvas, audio, video, form controls and template likewise;
  • event-handler attributes (on…) are dropped and reported; javascript:, vbscript: and data: link targets are removed (the link text is kept);
  • <meta http-equiv="refresh">, external style sheets and <base> are ignored; nothing is ever fetched: images come from data: URIs or from the caller's resourceLoader, which only ever receives relative paths without ..;
  • SVG images are refused (unsupported format), also as data: URIs;
  • elements hidden with hidden or display: none are not imported (reported);
  • nesting deeper than 256 open elements is flattened (reported); tables are cut at 256 columns and 1 000 000 cells (reported).

Known limits of the importer:

  • style sheets are not read, only inline style attributes (reported as Style sheets): a page whose formatting lives in CSS classes keeps its text and structure, not its fonts and colours;
  • misnested formatting is closed at the first end tag (<b>a<i>b</b>c</i>: c is not italic); the HTML "adoption agency" algorithm is not implemented;
  • the character encoding comes from the BOM, then UTF-8 validity, then a Windows-1252 fallback; <meta charset> is not read;
  • nested tables are flattened into the enclosing cell (reported); MathML without a TeX annotation becomes plain text (reported); formulas a producer already rendered as HTML have no LaTeX to recover (reported);
  • element ids become bookmarks only when a link of the document points at them (explicit <a id> anchors are always kept).

# Export

Standalone HTML5: one file, no external resource, no script. lang on <html>, metadata in <head>, an embedded style sheet derived from the document's StyleSheet (body, headings, title, subtitle, quote, code, caption, abstract, tables, custom paragraph styles as .ps-… classes), direct formatting as inline styles, images as data: URIs with alt, width and height, heading ids (pandoc-style slugs, or the heading's bookmark), tables with thead, colspan, rowspan, column widths and shading, footnotes and endnotes as lists in section.footnotes / section.endnotes with back links.

Equations: ExportOptions.mathML (LaTeX, display) → MathML markup, supplied by the caller (the DOCX engineer's MathConverter.mathML is the intended implementation; the two have not been run together). The markup is wrapped in <span class="math …" data-latex="…">, so the LaTeX source survives whatever the MathML looks like. Without a converter, or when it returns nil for an equation, the span holds the LaTeX source and a warning is issued.

Round trip (tested on the rich fixture): title, headings, direct character and paragraph formatting, links, images with their size, equations, footnotes and endnotes (as such), citations with locators, lists, table spans and shading, page breaks, the contents / figures / tables blocks and the bibliography block come back; a second generation is identical to the first. Not read back from our own output: custom paragraph styles (they are CSS classes), tab stops, paragraph borders, table borders.

# Verified with pandoc 3.9.0.2 and textutil

  • pandoc parses our HTML export: block sequence, list depth, inline node types, link target, code, and every word of the document in order.
  • We import pandoc's standalone HTML (-s) with --mathjax, --mathml and the default math rendering: same model as the Markdown source, plus the title block; with the default rendering the formulas come back as text because the LaTeX is not in the file.
  • textutil -convert txt on our HTML yields the same words in the same order.
  • We import the HTML textutil writes from our RTF: text, table and title are found; its formatting is in a <style> block, which we do not read.

# RTF

Reading and writing go through AppKit's NSAttributedString RTF codec, which is local (no network, no WebKit). Two things are added around it because the codec ignores them in plain RTF:

  • Pictures. \pict groups (PNG and JPEG, including Word's {\*\shppict …}{\nonshppict …} pairs, hex or \bin data, \picwgoal / \pichgoal / \picscale) are lifted out of the bytes before the codec runs and spliced back in after it writes. The bytes of an image survive a round trip unchanged. WMF, EMF and PICT pictures are dropped with a warning. Other image types in the model are converted to PNG on export.
  • Colours. A plain \red\green\blue colour table entry is taken as sRGB, as every other RTF producer means it, instead of being colour-matched from "generic RGB" (which turned pure red into 255/38/0).

Import maps: font family and size (the dominant ones become the document defaults, runs record only what differs), bold, italic, underline, strikethrough, text colour, background colour → highlight, super / subscript, baseline offset, kerning → tracking, links, alignment, indents (left, right, first line), spacing before / after, line spacing (multiple, exact, at least), tab stops, writing direction, lists (NSTextList → ListItem, number format and start kept, nesting to 9 levels), tables (NSTextTable blocks → Table with column and row spans, cell shading, vertical alignment, widths) when the cells tile a regular grid — otherwise paragraphs with a warning — page breaks, and title / author / subject / keywords / paper size / margins.

What RTF does not carry through this codec, in either direction: named styles (everything is direct formatting; headings come back as formatted body paragraphs), headers and footers, footnotes as footnotes, comments, tracked changes, equations, bookmarks, table header rows, cell borders, paragraph borders and shading, image alternative text, floating placement, sections and columns. On export, notes are appended after a separator, equations are written as LaTeX text, and each of these losses is a warning.

RTFD (RTF with attachments, a directory or its serialised form) is out of scope: it is refused with a message telling the user to save as RTF or DOCX.

# Verified

  • A hand-written RTF in the style of other producers (font table, colour table, \b \i \ul \strike \super \sub, \cf, \cb, \qc \qj \qr, \li \ri \fi, \sb \sa \sl, \tx \tqr, \'e9, \u escapes with a surrogate pair, \line, \page, \info, A4 paper and margins) imports with the expected text, formatting and metadata.
  • A hand-written RTF with a Word-style \shppict / \nonshppict picture pair, a \picprop group, scaling, a WMF-only picture and a truncated JPEG.
  • Round trip of the rich fixture: text, resolved character formatting of a run with font, size, colour with alpha, underline, strike, superscript, highlight and tracking; resolved paragraph formatting; lists; table spans, shading and vertical alignment; both images byte-identical; metadata.
  • textutil -convert txt on our RTF gives the expected lines; textutil -info shows our title; we import the RTF textutil writes from our HTML (text, bold, link, table, list).

Not verified: RTF written by Microsoft Word, LibreOffice or Pages. None of them is available on the build machine, and GUI applications are not launched by the tests. The Word-style constructs above were written by hand from the RTF specification. textutil uses the same system codec as we do, so it is an independent consumer of our pictures-free output but not an independent RTF implementation.

# Plain text

Import: byte-order mark first (UTF-8, UTF-16 LE / BE, UTF-32 LE / BE); then strict UTF-8; BOM-less UTF-16 recognised by the position of its zero bytes; mostly valid UTF-8 with a few damaged bytes is repaired with U+FFFD; otherwise Windows-1252, then ISO-8859-1 for the five bytes Windows-1252 leaves undefined. Any guess or repair is a warning that names the encoding. CR LF, CR and LF all end a line; each line is a paragraph (blank lines are empty paragraphs); a form feed is a page break; other control characters and U+FFFC are removed or replaced, with a warning.

Export: UTF-8 without BOM, one line per paragraph, U+2028 as a line break, list labels and heading numbers as text (four spaces per list level), table rows as tab-separated lines, page and section breaks as form feeds, notes appended as [n] text, equations as LaTeX, images as [alt text].

A plain document (text, tabs, blank lines, page breaks) round-trips byte for byte.

# PDF (import only)

PDFTextImporter uses PDFKit to read the text layer. The result always carries an approximated warning stating that layout, fonts, images, tables, headers and footers are not reconstructed.

Paragraphs are rebuilt per page from the lines PDFKit reports and their position: a new paragraph starts after a vertical gap clearly larger than the document's usual line pitch, when the text jumps back up the page, on a first-line indent, and after a short line that ends a sentence. A word hyphenated at a line end is joined when the next line starts with a lower-case letter (counted in a warning). Pages are separated by page breaks. Title, author, subject, keywords and the paper size of the first page are read.

Refused with an explicit error: data that is not a PDF, a password-protected PDF (the empty password is tried), a PDF whose owner forbids text extraction, and a PDF with no text on any page ("no text layer… run OCR first"). Pages without text in an otherwise readable PDF are counted in a warning. Of the paragraph rules, the vertical-gap rule, the short-line rule and hyphen joining are covered by tests; the first-line-indent and column rules, and the refusal of copy-restricted PDFs, are implemented but have no test.

Verified only with PDFs generated in the tests (Core Graphics + Core Text: single column, one font). Not verified: multi-column layouts, tables, right-to-left text, PDFs from Word, LaTeX or scanners with an OCR layer. The heuristics will misjudge some of these; this import is a way to get the words out, not a conversion.

# Safety limits

Limit Value Effect
Markdown, HTML, plain text source 64 MiB FormatError.tooLarge
RTF 128 MiB FormatError.tooLarge
PDF 512 MiB, 20 000 pages FormatError.tooLarge
.orixdoc 1 GiB, plus ZipLimits (entries, entry size, total size, compression ratio) FormatError.tooLarge
One image 64 MiB skipped, warning
Markdown block nesting 32 containers deeper content kept as text, warning
HTML open elements 256 deeper elements flattened, warning
List levels 9 deeper items flattened, warning
Table size (Markdown, HTML, RTF) 256 columns, 1 000 000 cells excess dropped or the table read as paragraphs, warning

The Markdown block parser is recursive within its bound and runs on its own 16 MiB stack, so its depth does not depend on the caller's thread. Tested: 10 000 nested block quotes, 2 000 lines of increasing quote depth, 1 000 nested list levels in Markdown (each level adds two columns of indentation, so 10 000 levels would be a 100 MB source, above the size limit: not tested), 10 000 nested ul/li, blockquote, div, b/i and table elements in HTML, 20 000-fold repetitions of delimiter, bracket, backtick and entity patterns, 500 × 40 tables, invalid UTF-8 of several kinds, random bytes fed to every importer, and randomly assembled markup checked against the model's invariants (inline objects, list ids and levels, rectangular tables, no dangling note or image, native round trip).

The entry points named importDocument on each importer never throw: a failure becomes an empty document with one dropped warning. read on each importer and DocumentIO.importDocument throw instead.

Files main
Menu

Theme