Format compatibility (DocFormats)
This page covers the formats handled by Packages/ORIXDocsKit/Sources/DocFormats: the native .orixdoc container, Markdown, HTML, RTF, plain text and PDF text import.
DOCX is documented with the DocDOCX module.
Nothing here is a benchmark or a conformance score. Every statement below is backed by a test in Tests/DocFormatsTests or is listed as not verified.
How to read the tables
- kept — comes back identical after export + import (round trip), or is read faithfully on import;
- approximated — content kept, form changed; the user gets an
approximatedwarning; - dropped — not carried; the user gets a
droppedwarning (exports) or the content is absent from the source format (imports); DocumentIO.preflight(_:format:)returns the export warnings before exporting, from the document alone. The exporters build their warnings from the same table, plus what they only discover while writing (a missing font, a missing image).
Summary
| .orixdoc | Markdown | HTML | RTF | Plain text | ||
|---|---|---|---|---|---|---|
| Import | yes | yes | yes | yes | yes | text only |
| Export | yes | yes | yes | yes | yes | no (DocLayout publishes PDF) |
| Text, Unicode | kept | kept | kept | kept up to canonical equivalence ¹ | kept | as extracted |
| Paragraph styles | kept | headings, quote, code, list | headings, title, subtitle, abstract, caption, quote, code, list | flattened to direct formatting | dropped | dropped |
| Bold / italic / strike | kept | kept | kept | kept | dropped | dropped |
| Underline, sub/superscript, highlight | kept | inline HTML (<u>, <sub>, <sup>, <mark>), colour of highlight dropped |
kept | kept | dropped | dropped |
| Fonts, sizes, colours | kept | dropped | kept (inline styles) | kept, fonts substituted when not installed ² | dropped | dropped |
| Alignment, indents, spacing | kept | dropped (table column alignment kept) | kept (inline styles) | kept | dropped | dropped |
| Tabs stops | kept | dropped | dropped (tab characters kept) | kept | tab characters kept | dropped |
| Links | kept | kept | kept | kept | text only | dropped |
| Lists, nesting, start number | kept | kept (9 levels; number formats become digits) | kept (start, type) |
kept | labels written as text | dropped |
| Tables | kept | pipe tables; merged cells become separate cells | kept with colspan / rowspan, shading |
kept with spans and shading; header-row flag dropped | tab-separated lines | dropped |
| Images | kept | kept (data URI or caller-supplied path); display size dropped | kept (data URI), size kept | PNG / JPEG kept, others converted to PNG; alt text dropped | alt text in brackets | dropped |
| Floating placement, crop, rotation | kept | inline | CSS float; crop and rotation dropped | inline | — | — |
| Shapes | kept | text only | text only | text only | text only | — |
| Equations | kept | kept ($…$, $$…$$) |
kept (data-latex; MathML when a converter is supplied) |
LaTeX source as text | LaTeX source as text | — |
| Footnotes | kept | kept | kept | appended at the end, marks as superscript text | appended at the end | — |
| Endnotes | kept | written as footnotes | kept | appended | appended | — |
| Citations | kept | kept ([@key, p. 4]) or formatted text |
kept (data-cite-items) + displayed text |
text | text | — |
| Bibliography block | kept | ::: {#refs} placeholder, or formatted entries |
kept as a block, entries when a formatter is supplied | formatted entries or nothing | formatted entries or nothing | — |
| Table of contents, lists of figures | kept | static list of links | kept as a block (<nav class="toc">) |
static text | static text | — |
| Captions, sequence fields | kept | resolved text | resolved text | resolved text | resolved text | — |
| Cross-references | kept | resolved text (linked when the target is a heading) | resolved text, linked | resolved text | resolved text | — |
| Page number / page count / running-heading fields | kept | left empty | left empty | left empty | left empty | — |
| Bookmarks | kept | kept ({#id} on headings, <a id> elsewhere) |
kept | dropped | dropped | — |
| Comments | kept | dropped | dropped | dropped | dropped | — |
| Tracked changes | kept | resolved (accepted, or rejected with acceptChanges = false) |
resolved | resolved | resolved | — |
| Page breaks | kept | dropped | CSS print break, read back | kept | form feed | kept between pages |
| Sections, page setup, columns | kept | dropped | dropped | paper size and margins only | dropped | paper size of page 1 |
| Headers and footers | kept | dropped | dropped | dropped | dropped | dropped |
| Metadata | kept | title, author, subject, keywords, language (front matter) | title, author, description, keywords, language | title, author, subject, keywords | dropped | title, author, subject, keywords |
| Preserved DOCX parts | kept | dropped | dropped | dropped | dropped | — |
¹ The system RTF codec writes precomposed characters where Windows-1252 has them (e + U+0301 comes back as é) and drops the bidirectional embedding controls U+202A…U+202E. The text is canonically equivalent, not always identical.
² Families with no installed named face (the system fonts "New York", "SF Pro Display", "SF Mono") are written as Times New Roman or Helvetica, with a warning.
Native .orixdoc
Lossless: the round trip of a document using every Block case, every Inline case, assets, notes, comments (with replies), tracked insertions / deletions / format changes, bibliography, sections and preserved parts is compared with ==.
Layout, integrity checks, recovery and migrations are described in docs/architecture/persistence.md.
Verified by tests: byte flips at the start, middle and end of each kind of entry (document, manifest, mimetype, asset, preserved part) are detected by read and by validate; a tampered asset inside a well-formed ZIP is caught by the SHA-256; truncation at several lengths; wrong mimetype; mimetype not first; missing manifest; manifest entry missing from the archive; future schema and container versions; unknown extra entries; recover on a damaged asset, a damaged manifest and a truncated file; the migration pipeline with an injected table.
Not verified: files larger than a few megabytes; behaviour at the 1 GiB limit (the limit is a comparison on the byte count, tested as such, not with a gigabyte of data).
Markdown
Parser and writer are our own (MarkdownImporter, MarkdownInline, MarkdownExporter); the dialect is CommonMark with the GFM and pandoc extensions listed here.
Import
| Construct | Result |
|---|---|
| ATX and setext headings | Heading1…Heading6; a pandoc attribute {#id} becomes a bookmark |
| Paragraphs, soft breaks | soft break → space |
Hard breaks (two spaces, backslash, <br>) |
U+2028 |
*em*, **strong**, _/__ (CommonMark delimiter rules) |
direct italic / bold |
~~strike~~, ~sub~, ^sup^, <u>, <sub>, <sup>, <mark> |
direct formatting |
`code` |
character style InlineCode |
Fenced (```, ~~~) and indented code |
one Code paragraph per block, lines separated by U+2028; the language label is dropped (the model has nowhere to store it) |
| Block quotes | Quote paragraphs; nested quotes become indented quotes; lists, code, headings and tables inside a quote lose the quote formatting (warning) |
| Lists | ListItem levels 0…8, one ListDefinition per top-level list, start number kept; deeper nesting is flattened to level 8 with a warning; continuation paragraphs are indented paragraphs |
| GFM task items | text prefixed with ☐ / ☑ |
| GFM pipe tables, pandoc simple tables | Table, first row is the header, column alignment on the cell paragraphs |
---, ***, ___ |
an empty paragraph with a bottom border (the model has no rule block) |
Links, reference links, autolinks, bare http(s):// URLs |
CharProps.link + Hyperlink style; link titles dropped |
| Images | data: URIs and relative paths through resourceLoader; remote URLs are never fetched: alt text + warning; paths that are absolute, contain .. or a scheme are refused |
Footnotes [^id] |
Note (multi-paragraph definitions supported); unused definitions dropped with a warning; references without definition stay literal |
| YAML front matter | title, author, subject / description, keywords / tags, lang / language; other keys reported |
$…$, $$…$$ |
Equation (inline / display); a $$ block alone in a paragraph is a display-equation paragraph |
[@key, p. 4; -@other] |
Citation with prefix, locator / suffix, suppress-author |
::: {#refs} |
the bibliography block; other fenced divs are transparent |
| Character references | decoded (HTML 4 names and numeric forms) |
Not supported on import: pandoc grid and multiline tables, definition lists, inline footnotes ^[…], bare @key citations outside brackets, raw HTML blocks (kept as literal text, with a warning), link titles, image sizes, emoji shortcodes.
Export
The exporter writes pandoc-flavoured Markdown. Text is escaped so that it reads back as text (including :// and www., so a string that merely looks like a URL does not come back as a link).
Emphasis markers are always placed against a word; when CommonMark's flanking rules would otherwise misread a boundary (bold immediately followed by italic, a closing marker after punctuation and before a letter), an empty HTML comment <!----> is inserted. It is invisible when rendered and ignored on import.
Approximations specific to the exporter: the Title paragraph becomes a level 1 heading; heading levels 7–9 become level 6; automatic heading numbers are written as text; consecutive Code paragraphs form one fenced block; a table always gets a header row; empty paragraphs are not written; list number formats (letters, roman numerals) become digits.
A second export of an imported export is byte-identical (tested on the rich fixture).
Verified with pandoc 3.9.0.2
- pandoc parses our Markdown export of a document with headings, emphasis, strikethrough, code, link, footnote, inline and display math, three list levels, an ordered list starting at 3, a quote, a code block, a table, a rule, non-Latin text and a hard break: the block sequence of its JSON AST, the inline node types, the link target, the LaTeX of both equations, the code, the note count and the full word sequence match the model. No raw HTML and no citation node appears, i.e. nothing we escaped was read as markup.
pandoc -f gfmreads the same export (without math) with the same structure.- We import what pandoc writes from one source in four dialects (
markdown,gfm,commonmark_x,markdown_strict+ extensions) and get the same model as from the source, with these documented differences: GFM has no heading ids (the internal link keeps its target, the heading gets no anchor) and writes math in GitHub's forms, which stay code for us. - Our export, rewritten by pandoc and imported again, gives the same structure as importing our export directly.
Skipped when pandoc is not installed.
HTML
Import
Our own tokenizer and tree builder (HTMLTokenizer, HTMLImporter). NSAttributedString's HTML import is not used: it runs WebKit and can load remote resources.
Handled: p, div, h1–h6, br, hr, b/strong, i/em/cite/dfn/var, u/ins, s/strike/del, sub, sup, code/kbd/samp/tt, pre, blockquote, a, mark, q, font, center, ul/ol/li (nested, start, type), dl/dt/dd, table/thead/tbody/tfoot/tr/th/td/caption with colspan/rowspan, img, figure/figcaption, title, meta (author, keywords, description), lang, dir, align, bgcolor, character references.
Inline style subset: font-family, font-size, color, background-color, text-align, font-weight, font-style, text-decoration, vertical-align (super / sub), font-variant: small-caps, text-transform: uppercase, white-space: pre*, margin-left / margin-right / margin-top / margin-bottom, text-indent, line-height, direction, display: none, page-break properties.
Recognised conventions: pandoc and ORIX footnotes (a.footnote-ref + section.footnotes), span.math with a data-latex attribute or delimited TeX, MathML with a TeX annotation, span.citation with data-cite-items / data-cites, nav.toc, section.orix-bibliography, .page-break, h1.title, p.subtitle, p.abstract, p.caption.
Security behaviour (all tested):
<script>and<style>content is skipped by the tokenizer and never interpreted; scripts are reported as dropped;iframe,frame,object,embed,appletare removed with their content;svg,canvas,audio,video, form controls andtemplatelikewise;- event-handler attributes (
on…) are dropped and reported;javascript:,vbscript:anddata:link targets are removed (the link text is kept); <meta http-equiv="refresh">, external style sheets and<base>are ignored; nothing is ever fetched: images come fromdata:URIs or from the caller'sresourceLoader, which only ever receives relative paths without..;- SVG images are refused (unsupported format), also as
data:URIs; - elements hidden with
hiddenordisplay: noneare not imported (reported); - nesting deeper than 256 open elements is flattened (reported); tables are cut at 256 columns and 1 000 000 cells (reported).
Known limits of the importer:
- style sheets are not read, only inline
styleattributes (reported asStyle sheets): a page whose formatting lives in CSS classes keeps its text and structure, not its fonts and colours; - misnested formatting is closed at the first end tag (
<b>a<i>b</b>c</i>:cis not italic); the HTML "adoption agency" algorithm is not implemented; - the character encoding comes from the BOM, then UTF-8 validity, then a Windows-1252 fallback;
<meta charset>is not read; - nested tables are flattened into the enclosing cell (reported); MathML without a TeX annotation becomes plain text (reported); formulas a producer already rendered as HTML have no LaTeX to recover (reported);
- element
ids become bookmarks only when a link of the document points at them (explicit<a id>anchors are always kept).
Export
Standalone HTML5: one file, no external resource, no script. lang on <html>, metadata in <head>, an embedded style sheet derived from the document's StyleSheet (body, headings, title, subtitle, quote, code, caption, abstract, tables, custom paragraph styles as .ps-… classes), direct formatting as inline styles, images as data: URIs with alt, width and height, heading ids (pandoc-style slugs, or the heading's bookmark), tables with thead, colspan, rowspan, column widths and shading, footnotes and endnotes as lists in section.footnotes / section.endnotes with back links.
Equations: ExportOptions.mathML (LaTeX, display) → MathML markup, supplied by the caller (the DOCX engineer's MathConverter.mathML is the intended implementation; the two have not been run together). The markup is wrapped in <span class="math …" data-latex="…">, so the LaTeX source survives whatever the MathML looks like. Without a converter, or when it returns nil for an equation, the span holds the LaTeX source and a warning is issued.
Round trip (tested on the rich fixture): title, headings, direct character and paragraph formatting, links, images with their size, equations, footnotes and endnotes (as such), citations with locators, lists, table spans and shading, page breaks, the contents / figures / tables blocks and the bibliography block come back; a second generation is identical to the first. Not read back from our own output: custom paragraph styles (they are CSS classes), tab stops, paragraph borders, table borders.
Verified with pandoc 3.9.0.2 and textutil
- pandoc parses our HTML export: block sequence, list depth, inline node types, link target, code, and every word of the document in order.
- We import pandoc's standalone HTML (
-s) with--mathjax,--mathmland the default math rendering: same model as the Markdown source, plus the title block; with the default rendering the formulas come back as text because the LaTeX is not in the file. textutil -convert txton our HTML yields the same words in the same order.- We import the HTML
textutilwrites from our RTF: text, table and title are found; its formatting is in a<style>block, which we do not read.
RTF
Reading and writing go through AppKit's NSAttributedString RTF codec, which is local (no network, no WebKit).
Two things are added around it because the codec ignores them in plain RTF:
- Pictures.
\pictgroups (PNG and JPEG, including Word's{\*\shppict …}{\nonshppict …}pairs, hex or\bindata,\picwgoal/\pichgoal/\picscale) are lifted out of the bytes before the codec runs and spliced back in after it writes. The bytes of an image survive a round trip unchanged. WMF, EMF and PICT pictures are dropped with a warning. Other image types in the model are converted to PNG on export. - Colours. A plain
\red\green\bluecolour table entry is taken as sRGB, as every other RTF producer means it, instead of being colour-matched from "generic RGB" (which turned pure red into 255/38/0).
Import maps: font family and size (the dominant ones become the document defaults, runs record only what differs), bold, italic, underline, strikethrough, text colour, background colour → highlight, super / subscript, baseline offset, kerning → tracking, links, alignment, indents (left, right, first line), spacing before / after, line spacing (multiple, exact, at least), tab stops, writing direction, lists (NSTextList → ListItem, number format and start kept, nesting to 9 levels), tables (NSTextTable blocks → Table with column and row spans, cell shading, vertical alignment, widths) when the cells tile a regular grid — otherwise paragraphs with a warning — page breaks, and title / author / subject / keywords / paper size / margins.
What RTF does not carry through this codec, in either direction: named styles (everything is direct formatting; headings come back as formatted body paragraphs), headers and footers, footnotes as footnotes, comments, tracked changes, equations, bookmarks, table header rows, cell borders, paragraph borders and shading, image alternative text, floating placement, sections and columns. On export, notes are appended after a separator, equations are written as LaTeX text, and each of these losses is a warning.
RTFD (RTF with attachments, a directory or its serialised form) is out of scope: it is refused with a message telling the user to save as RTF or DOCX.
Verified
- A hand-written RTF in the style of other producers (font table, colour table,
\b \i \ul \strike \super \sub,\cf,\cb,\qc \qj \qr,\li \ri \fi,\sb \sa \sl,\tx \tqr,\'e9,\uescapes with a surrogate pair,\line,\page,\info, A4 paper and margins) imports with the expected text, formatting and metadata. - A hand-written RTF with a Word-style
\shppict/\nonshppictpicture pair, a\picpropgroup, scaling, a WMF-only picture and a truncated JPEG. - Round trip of the rich fixture: text, resolved character formatting of a run with font, size, colour with alpha, underline, strike, superscript, highlight and tracking; resolved paragraph formatting; lists; table spans, shading and vertical alignment; both images byte-identical; metadata.
textutil -convert txton our RTF gives the expected lines;textutil -infoshows our title; we import the RTFtextutilwrites from our HTML (text, bold, link, table, list).
Not verified: RTF written by Microsoft Word, LibreOffice or Pages. None of them is available on the build machine, and GUI applications are not launched by the tests. The Word-style constructs above were written by hand from the RTF specification. textutil uses the same system codec as we do, so it is an independent consumer of our pictures-free output but not an independent RTF implementation.
Plain text
Import: byte-order mark first (UTF-8, UTF-16 LE / BE, UTF-32 LE / BE); then strict UTF-8; BOM-less UTF-16 recognised by the position of its zero bytes; mostly valid UTF-8 with a few damaged bytes is repaired with U+FFFD; otherwise Windows-1252, then ISO-8859-1 for the five bytes Windows-1252 leaves undefined. Any guess or repair is a warning that names the encoding. CR LF, CR and LF all end a line; each line is a paragraph (blank lines are empty paragraphs); a form feed is a page break; other control characters and U+FFFC are removed or replaced, with a warning.
Export: UTF-8 without BOM, one line per paragraph, U+2028 as a line break, list labels and heading numbers as text (four spaces per list level), table rows as tab-separated lines, page and section breaks as form feeds, notes appended as [n] text, equations as LaTeX, images as [alt text].
A plain document (text, tabs, blank lines, page breaks) round-trips byte for byte.
PDF (import only)
PDFTextImporter uses PDFKit to read the text layer. The result always carries an approximated warning stating that layout, fonts, images, tables, headers and footers are not reconstructed.
Paragraphs are rebuilt per page from the lines PDFKit reports and their position: a new paragraph starts after a vertical gap clearly larger than the document's usual line pitch, when the text jumps back up the page, on a first-line indent, and after a short line that ends a sentence. A word hyphenated at a line end is joined when the next line starts with a lower-case letter (counted in a warning). Pages are separated by page breaks. Title, author, subject, keywords and the paper size of the first page are read.
Refused with an explicit error: data that is not a PDF, a password-protected PDF (the empty password is tried), a PDF whose owner forbids text extraction, and a PDF with no text on any page ("no text layer… run OCR first"). Pages without text in an otherwise readable PDF are counted in a warning. Of the paragraph rules, the vertical-gap rule, the short-line rule and hyphen joining are covered by tests; the first-line-indent and column rules, and the refusal of copy-restricted PDFs, are implemented but have no test.
Verified only with PDFs generated in the tests (Core Graphics + Core Text: single column, one font). Not verified: multi-column layouts, tables, right-to-left text, PDFs from Word, LaTeX or scanners with an OCR layer. The heuristics will misjudge some of these; this import is a way to get the words out, not a conversion.
Safety limits
| Limit | Value | Effect |
|---|---|---|
| Markdown, HTML, plain text source | 64 MiB | FormatError.tooLarge |
| RTF | 128 MiB | FormatError.tooLarge |
| 512 MiB, 20 000 pages | FormatError.tooLarge |
|
.orixdoc |
1 GiB, plus ZipLimits (entries, entry size, total size, compression ratio) |
FormatError.tooLarge |
| One image | 64 MiB | skipped, warning |
| Markdown block nesting | 32 containers | deeper content kept as text, warning |
| HTML open elements | 256 | deeper elements flattened, warning |
| List levels | 9 | deeper items flattened, warning |
| Table size (Markdown, HTML, RTF) | 256 columns, 1 000 000 cells | excess dropped or the table read as paragraphs, warning |
The Markdown block parser is recursive within its bound and runs on its own 16 MiB stack, so its depth does not depend on the caller's thread.
Tested: 10 000 nested block quotes, 2 000 lines of increasing quote depth, 1 000 nested list levels in Markdown (each level adds two columns of indentation, so 10 000 levels would be a 100 MB source, above the size limit: not tested), 10 000 nested ul/li, blockquote, div, b/i and table elements in HTML, 20 000-fold repetitions of delimiter, bracket, backtick and entity patterns, 500 × 40 tables, invalid UTF-8 of several kinds, random bytes fed to every importer, and randomly assembled markup checked against the model's invariants (inline objects, list ids and levels, rectangular tables, no dangling note or image, native round trip).
The entry points named importDocument on each importer never throw: a failure becomes an empty document with one dropped warning. read on each importer and DocumentIO.importDocument throw instead.