SPB Git

Repository Public

text-engine.md 28.3 KB

# Text engine

The editor is not an NSTextView. It is a value-type model (DocModel), an editing engine that transforms that model (DocumentEditor), and a custom NSView that turns keyboard, mouse and pasteboard events into editing operations (EditorCanvasView). Why a custom view rather than TextKit is argued in technology-evaluation.md, section 1; the model itself is summarised in overview.md.

Layer Code
Model Packages/ORIXDocsKit/Sources/DocModel/Document.swift, Properties.swift, Styles.swift, Numbering.swift
Text primitives, addressing ParagraphText.swift, Position.swift
Editing operations Editor.swift, EditorTables.swift, EditorReview.swift
Session (undo, relayout, typing rules) App/Sources/Document/DocumentSession.swift, Commands.swift
Input App/Sources/Canvas/EditorCanvasView.swift, EditorCanvasView+Keyboard.swift, EditorCanvasView+Mouse.swift
Tests Tests/DocModelTests/EditingTests.swift, ReviewAndTableTests.swift; App/Sources/App/SelfTest.swift (see testing.md)

# 1. A value-type model

Every model type is a struct or enum that is Codable, Hashable and Sendable (Document.swift). Consequences used throughout the code:

  • Undo is a stack of values. DocumentSession.perform keeps the DocumentModel it had before the change; arrays are copy-on-write, so the snapshot shares the storage of everything the edit did not touch.
  • Background saving needs no locking of the model. ModelStore hands a copy to the saving thread (persistence.md).
  • Layout caching by equality. LayoutCache compares the stored Paragraph with the current one (==) to decide whether a paragraph must be typeset again (layout-engine.md).
  • Input-method composition rebuilds a provisional document from a saved value on every keystroke and throws it away on commit (section 9).
classDiagram
    class DocumentModel {
        meta
        settings
        styles
        lists
        pageSetup
        body
        notes
        comments
        bibliography
        assets
    }
    class Block {
        <<enum>>
        paragraph
        table
        pageBreak
        sectionBreak
        toc
        bibliography
    }
    class Paragraph {
        id
        style
        props
        runs
        list
        markRevision
    }
    class Run {
        text
        props
        charStyle
        inline
        revision
        formatRevision
        comments
    }
    class Inline {
        <<enum>>
        image
        shape
        equation
        note
        citation
        crossRef
        field
        bookmark
    }
    class Table {
        columnWidths
        rows
        style
    }
    class TableCell {
        paragraphs
        colSpan
        vMerge
        formula
    }
    class NoteContent {
        kind
        paragraphs
    }
    class PageSetup {
        paper
        margins
        columns
        header
        footer
    }
    DocumentModel "1" --> "*" Block : body
    DocumentModel "1" --> "*" NoteContent : notes
    DocumentModel "1" --> "1" PageSetup : first section
    Block --> Paragraph
    Block --> Table
    Block --> PageSetup : sectionBreak carries one
    Paragraph "1" --> "*" Run
    Run --> Inline : optional
    Table "1" --> "*" TableCell : rows of cells
    TableCell "1" --> "*" Paragraph
    NoteContent "1" --> "*" Paragraph
    PageSetup --> Paragraph : header and footer variants

In the diagram NoteContent stands for the model type Note (renamed only because note is a reserved word in the diagram language).

# 2. Runs and inline objects

A Paragraph is an array of Run. A run is a string plus optional direct formatting (CharProps, every field optional = inherit), an optional character style, an optional tracked Revision, an optional FormatRevision and the ids of the comments anchored on it.

  • An inline object is a run whose text is the single UTF-16 unit U+FFFC (objectReplacementCharacter) and whose inline holds the payload (Run.init(inline:)). Images and shapes can be .inline or .floating (Placement); the other kinds are always inline.
  • A forced line break is U+2028 in the text (insertLineBreak); a tab is \t.
  • Paragraph.normalize() drops empty runs and merges neighbours for which Run.canMerge(with:) holds (same props, style, revision, format revision and comments, no inline object). Every mutating primitive ends with it, so two equal paragraphs have equal run arrays.
  • plainText is the concatenation of all runs, deleted text included; visibleText is the text with changes accepted and inline objects removed.

# 3. UTF-16 offsets and composed-character boundaries

Every offset in the model is a UTF-16 code unit offset (Run.length, Paragraph.length), the convention of NSRange, NSTextInputClient and Core Text. An offset may never fall inside a surrogate pair, a ZWJ emoji sequence, or between a base letter and its combining marks. This is enforced where offsets are produced, not where they are consumed:

Function (ParagraphText.swift, Editor.swift) Role
String.composedBoundary(atOrBefore:), (after:), (before:) Wrap NSString.rangeOfComposedCharacterSequence(at:).
DocumentEditor.position(after:in:), position(before:in:) Left/Right arrows: one composed character at a time, then across paragraphs, cells, rows and blocks.
DocumentEditor.clamp(_:in:preferEnd:), sanitizeSelection() Every selection set through DocumentSession.select is snapped back onto a boundary.
deleteBackward(_:), deleteForward(_:) Delete the whole composed character (or word).
String.wordBoundary(before:), (after:), wordRange(at:) Word movement and double-click; a surrogate half counts as a word character, and results are re-aligned with composedBoundary(atOrBefore:).
LayoutLine.offset(atX:) (DocLayout) Hit testing returns what CTLineGetStringIndexForPosition returns; TypesettingTests.complexScriptsLayOutAndMapBack checks these offsets sit on boundaries for Arabic, Hebrew, Japanese, Devanagari and emoji sequences.

Tests: UnicodeTests in EditingTests.swift (backspaceRemovesWholeComposedCharacters, caretMovementNeverLandsInsideACluster, clampSnapsOffsetsOntoBoundaries, splitInsideSurrogatePairIsImpossibleThroughTheEditor), and the self-test scenario "Unicode typing deletes whole characters".

Low-level primitives (Paragraph.splitRun(at:), delete(_:), transform(_:_:)) do not re-align offsets: the file header states that callers align them. Two callers pass ranges they did not align themselves: DocumentEditor.replace(_:with:) uses the range reported by NSString.range(of:) or NSRegularExpression, and insertText(_:replacementRange:) uses the range given by the input method.

# 4. Stories, paths, positions

  • A story (StoryID) is an independently editable flow: .body, .header(section:variant:), .footer(section:variant:), .note(id). DocumentModel.blocks(in:) / setBlocks(_:in:) read and write one; header, footer and note stories hold paragraphs only (other blocks are dropped by setBlocks).
  • A path (ParaPath) is a block index plus an optional CellPath (row, cell index in the row — not the grid column — and paragraph index in the cell).
  • A position (Position) is a path and an offset. In a paragraph the offset is a UTF-16 offset. On an atomic block (page/column break, section break, generated list, bibliography) or on a table taken as a whole, it is 0 (before) or 1 (after).
  • A selection (Selection) is a story, an anchor and a head; start/end order them with Position.<.
  • The editor works on a container: the story's block array, or the paragraphs of one cell presented as blocks (DocumentEditor.container, setContainer). Tables pasted into a cell are flattened to their paragraphs by setContainer — there are no nested tables.

# Selection resolution

DocumentEditor.resolve(_:_:) turns a selection into a SelectionTarget:

Situation Target
Anchor equals head .caret(position)
Both ends in the same container (the story, or the same cell) .range(start, end)
Both ends in the same table but in different cells .cells(block:rows:columns:) — the rectangle of rows and grid columns spanned by the two cells, merged cells widening it
One end inside a table, the other outside (or in another table) The end inside is replaced by the whole table (offset 0 if it is the start, 1 if it is the end): a selection that crosses a table edge always takes the entire table. Test: selectionFromInsideATableToOutsideCoversTheWholeTable.

selectedParagraphs() lists every paragraph touched with its selected range; formatting commands iterate over it.

# 5. Editing operations

All are mutating methods of DocumentEditor; tracking is doc.settings.trackChanges.

Operation Behaviour Under change tracking
insertText(_:) Replaces the selection; CR, CRLF and U+2029 become paragraph breaks. Typed text takes typingAttributes(at:): the formatting of the text on the left, without its link at a link's end; typingOverride (⌘B with a caret) wins. Runs get an insertion Revision. insertionRevision(in:at:) reuses the revision of an adjacent pending insertion by the same author, so a burst of typing is one change.
insertRuns(_:inheritAttributes:), insertInline(_:) Insert at the caret; a caret on an atomic block first gets an empty paragraph (ensureCaretInParagraph). Same as above.
insertParagraphBreak() Splits the paragraph. At the end of a paragraph the new one takes the style's next; at offset 0 the ids are swapped so the content keeps its id (cross-references target ids). Return on an empty list item lowers its level or leaves the list. The first half gets an inserted markRevision.
insertLineBreak() Inserts U+2028. Tracked insertion.
deleteSelection(caretAtEnd:) on .range deleteRange: the two end paragraphs are joined; an emptied first paragraph adopts the formatting of the text that follows; atomic blocks and tables wholly inside the range are removed. trackDelete marks runs as deleted and keeps the text; runs the same author had inserted and that are still pending are removed for good; an adjacent deletion by the same author is extended. Each paragraph mark inside the range gets a deletion markRevision, unless the author had inserted that mark, in which case the paragraphs are simply joined. Atomic blocks and tables wholly inside the range are removed without a trace.
deleteSelection on .cells Empties the selected cells and clears their formulas. Not tracked.
deleteBackward(_:) / deleteForward(_:) (character or word) Inside text: delete to the previous/next boundary. At offset 0: leave the list if the paragraph is a list item, else join with the previous paragraph, step into a preceding table (no deletion), or remove a preceding atomic block. At the end: join with the next paragraph or remove a following atomic block; nothing in front of a table. Text: trackDelete. Joining: the paragraph mark gets a deletion revision and the caret moves across it (unless the mark was the author's own insertion). Atomic blocks are removed untracked.
formatCharacters(_:), toggle(_:) With a caret: stored in typingOverride. With a selection: applied to every run of selectedParagraphs(). A FormatRevision holding the previous CharProps and character style is set on runs whose properties changed, that carry none yet, and that are not themselves insertions.
setCharacterStyle(_:), setLink(_:), clearCharacterFormatting() Direct changes on the selected runs. clearCharacterFormatting goes through formatCharacters; the style and link changes are not tracked.
formatParagraphs(_:), setParagraphStyle(_:), toggleList(_:), changeListLevel(by:), updateStyleFromSelection(_:) Paragraph properties, style, list membership. Not tracked.
insertBlock(_:) Splits the paragraph at the caret and inserts a table, break or generated block (body only, never inside a cell); a paragraph is guaranteed after a table and at the end of the body. Not tracked.
insertFragment(_:) Paste. Ids of paragraphs, tables, breaks and notes are regenerated; assets and list definitions are brought over; unknown styles fall back to Normal. A single-paragraph fragment is inserted inline (its paragraph style is not applied). In a cell, header, footer or note, tables are flattened and breaks dropped. One insertion revision on all pasted runs and paragraph marks (including the cells of a pasted table).
insertNote(_:), insertImage(…), updateInline(at:in:_:) Create the note or asset and insert the inline object; replace an inline object in place (resize, equation edit). Insertion is tracked; updateInline is not.
Table structure (EditorTables.swift): insertRow, insertColumn, deleteRows, deleteColumns, deleteTable, mergeCells, splitCell, formatCells, resizeColumn, sortRows, setCellFormula, recalculateTable Operate on the grid; vertical merges are kept consistent (deleteRows restarts a merge whose first row goes away, insertRow continues one). sortRows refuses tables with vertical merges. Not tracked.
replace(_:with:), replaceAll(_:with:options:) Delete + insert with the formatting of the first replaced character; replaceAll goes back to front. Tracked as a deletion and an insertion.
addComment(_:) and comment edits Append the comment id to the selected runs (the word around the caret when nothing is selected). Not a tracked change.
moveSection(startingAt:before:) Moves a heading and everything under it in the body. Not tracked.

After deletions pruneOrphans() removes notes whose reference is gone and comments with no anchor left.

What is not tracked, in one place: table structure and cell clearing, atomic blocks (inserted or deleted), paragraph formatting, styles and lists, character-style and link changes, object resizing, section moves, and the hyphen that the smart-dash rule replaces (section 10).

# 6. Accepting and rejecting changes

DocumentModel.trackedChanges() lists changes in reading order (body, headers and footers, notes); consecutive runs sharing a revision id form one entry. resolveChanges(accept:ids:) (EditorReview.swift) resolves all changes, or those whose revision id is in ids:

Change Accept Reject
Inserted run revision cleared run removed
Deleted run run removed revision cleared
Format revision cleared previous CharProps and character style restored
Inserted paragraph mark cleared paragraph joined with the next one
Deleted paragraph mark paragraph joined with the next one cleared
  • Joining continues while the next paragraph's own mark also resolves to a join. When the first paragraph is empty and the next is not, the result keeps the next paragraph (id and formatting); otherwise the first paragraph keeps its formatting and receives the runs. Joins never cross a table, an atomic block or a cell boundary (paragraphs are resolved in maximal sequences of adjacent paragraphs).
  • Whole table rows: a row disappears when every cell that is not a vertical-merge continuation is "wholly" doomed — each of its paragraphs has a mark and only runs that the operation removes (rejected insertion or accepted deletion). Whole columns are handled the same way when no row qualifies and no row has a horizontally merged cell. A table left without rows is dropped. The editor itself never creates such row or column revisions except by pasting a table while tracking; they also arrive from DOCX import and from comparison (review.md).
  • DocumentEditor.resolveChanges then runs pruneOrphans() and sanitizeDocument().

Tests: TrackChangesTests in ReviewAndTableTests.swift; self-test scenario "track changes: insert, delete, accept and reject".

# 7. Undo

DocumentSession.perform(_:coalesce:_:) is the single entry point for undoable edits:

  1. it copies (doc, selection), runs the closure on the editor, and
  2. registers an undo action that swaps the saved state back in and registers the inverse (so redo is the same mechanism), unless the step coalesces with the previous one;
  3. it calls documentChanged() (section 8).

Coalescing requires the same key as the previous step, less than 2.5 s between them, and an undo stack that is not empty. Keys in use: typing (DocumentSession.type), delete (single-character Backspace/Delete), resize-column, resize-object, font-size. Any selection change made through select, any undo or redo, and any step without a key (Return, paste, formatting) ends the burst.

registerUndo opens an explicit undo group when UndoManager.groupsByEvent is off, so each step is its own unit even when many events are processed in one run-loop pass (the self-test does this). Undo restores the selection as well as the document and clears typingOverride.

# 8. From key press to pixels

sequenceDiagram
    participant W as NSWindow
    participant C as EditorCanvasView
    participant IC as NSTextInputContext
    participant S as DocumentSession
    participant E as DocumentEditor
    participant P as Paginator
    W->>C: keyDown
    C->>IC: handleEvent
    alt text
        IC->>C: insertText
        C->>S: type
        S->>S: smart quotes and dashes
    else command
        IC->>C: doCommand
    end
    C->>S: perform
    S->>E: editing operation
    S->>S: register undo unless coalesced
    S->>S: store model for background saving
    S->>P: layout with cache and previous layout
    P-->>S: DocumentLayout
    S->>C: layoutChanged
    C->>C: recompute page frames, needsDisplay, scroll to caret
    C->>C: draw visible pages with PageRenderer, then overlays

documentChanged() also refreshes find matches and schedules the word count 0.35 s later. Drawing is described in layout-engine.md; draw(_:) only renders pages whose frame intersects the dirty rectangle, then overlays (header/footer/note frame, find matches, spelling, selection, object handle, composition underline, caret).

# 9. Keyboard input

EditorCanvasView conforms to NSTextInputClient; keyDown passes every event to inputContext.handleEvent.

What ranges mean. All NSRange values exchanged with the input system are UTF-16 ranges relative to the paragraph that holds the caret, not to the document:

Member Implementation
selectedRange() The selection inside the caret's paragraph; when the selection spans several paragraphs, a zero-length range at the head.
markedRange(), hasMarkedText() The composition range in that paragraph, or NSNotFound.
attributedSubstring(forProposedRange:actualRange:) Plain text of the paragraph, clamped (no attributes).
firstRect(forCharacterRange:actualRange:) The caret rectangle at the range's location in screen coordinates (the range length is ignored).
characterIndex(for:) Offset of the hit position inside the paragraph that was hit, within the story being edited.
validAttributesForMarkedText() Empty.
insertText(_:replacementRange:) Commits a composition, or selects replacementRange in the caret's paragraph first when it is valid. "\n"/"\r" insert a paragraph break. Plain typing goes through DocumentSession.type.

The optional protocol members are not implemented.

Marked text (input methods). On the first setMarkedText the view saves compositionBase (document and selection). Each update rebuilds an editor from that base with tracking switched off, deletes the selection, inserts the provisional text (newlines become spaces), records marked and shows the result with applyComposition — relayout without undo registration and without updating the model given to autosave. An empty marked string restores the base. On commit (insertText) the base is restored and the final text is inserted through perform, as one undo step with normal change tracking; smart quotes are not applied to composed text. unmarkText() (a key-binding command, a mouse click) keeps the composed document as one undo step. The marked range is underlined in the accent colour.

Key-binding commands handled in doCommand(by:) (anything else is ignored):

  • movement, each with its …AndModifySelection: form: moveLeft:/moveBackward:, moveRight:/moveForward:, moveWordLeft:/moveWordBackward:, moveWordRight:/moveWordForward:, moveUp:, moveDown:, moveToBeginningOfLine:/moveToLeftEndOfLine:, moveToEndOfLine:/moveToRightEndOfLine:, moveToBeginningOfParagraph:, moveToEndOfParagraph: (plus moveParagraphBackwardAndModifySelection:, moveParagraphForwardAndModifySelection:), moveToBeginningOfDocument:, moveToEndOfDocument:, pageUp:, pageDown:;
  • scrolling aliases: scrollToBeginningOfDocument:, scrollToEndOfDocument:, scrollPageUp:, scrollPageDown:, centerSelectionInVisibleArea:;
  • deletion: deleteBackward:, deleteForward:, deleteWordBackward:, deleteWordForward:, deleteToBeginningOfLine:, deleteToEndOfLine:, deleteToEndOfParagraph: (the last two both delete to the end of the visual line);
  • insertion: insertNewline:, insertNewlineIgnoringFieldEditor:, insertLineBreak:, insertTab:, insertTabIgnoringFieldEditor:, insertBacktab:;
  • cancelOperation: (leave a header, footer or note; else leave Focus mode) and selectAll:.

Tab moves to the next cell in a table (adding a row after the last cell and selecting the destination cell's content), indents a list item when the caret is at offset 0, and otherwise inserts a tab character. Up and Down keep a goal x position (goalX) and use DocumentLayout.verticalMove; Home/End use lineRange and remember upstream affinity (caretUpstream) so the caret can sit at the end of a wrapped line.

# 10. Smart quotes and dashes

DocumentSession.type(_:) applies them to a single typed character when the document settings ask for it (smartQuotes, smartDashes, both on by default):

  • "Opening" context: nothing before the caret in the paragraph, or the previous UTF-16 unit is white space or one of ( [ { « “ ‘ — – -.
  • " becomes “ / ”; when the language of the text (run language, else document language) starts with fr, it becomes « + no-break space, or no-break space + ».
  • ' becomes ‘ in an opening context (never in French), ’ otherwise.
  • - typed right after - with a caret replaces both by —, except when the current font family is that of the Code style. The first hyphen is deleted with tracking switched off.

There is no en-dash or ellipsis rule. Pasted text, dropped text and committed input-method text are inserted as they are.

# 11. Mouse

mouseDown (EditorCanvasView+Mouse.swift), in order: the resize handle of the selected object; a table column boundary (within 3 pt, DocumentLayout.columnBoundary); a floating object; then text hit testing with hit(at:clickCount:):

  • clicks stay in the story being edited; a note is entered with one click on it, a header or footer with a double click, and a double click in the body leaves a header or footer;
  • a click on an inline image, shape or equation selects it (a one-character range); a double click on an equation opens the equation sheet;
  • ⌘-click follows a link (#name jumps to a bookmark or block id; only http, https and mailto URLs are opened);
  • one click places the caret (Shift extends), two select the word (wordRange(at:)), three the paragraph;
  • dragging after a single click extends the selection with autoscroll; a drag that starts with a double or triple click does not extend the selection (dragState is only set to .selecting for a single click).

Dragging a column boundary or an object handle performs coalesced resizeColumn / updateInline steps.

# 12. Clipboard, drag and drop

copy(_:) writes four flavours: the native fragment (ai.spboucher.orix.docs.fragment, the JSON of DocumentFragment), plain text, and RTF and HTML produced by the regular exporters from a temporary document. copyFragment() leaves out deleted runs and strips revisions and comments; a cell selection is copied as a table.

paste(_:) takes the first flavour available in this order: native fragment → a file URL whose type is an image → RTF → HTML → PNG or TIFF data → plain text. "Paste and Match Style" (pasteAsPlainText(_:)) uses plain text only. cut(_:) is copy followed by deleteSelection().

The view accepts drops of file URLs (images and PDF files are inserted as images), PNG/TIFF data, and strings; the caret follows the pointer while dragging. RTF, HTML and the native fragment are not read from a drop, and the view is not a drag source: selected text cannot be dragged out or moved by dragging.

# 13. Spelling

SpellingCache (EditorCanvasView.swift) calls NSSpellChecker.shared.check(_:range:types:…) with the spelling type only, on the display text of a paragraph, and caches the result per paragraph id until that text changes (the cache is emptied when it exceeds 4,000 entries; paragraphs of 20,000 UTF-16 units or more are not checked). The check runs while drawing, for the visible pages, in the body and in the story being edited. The word that contains the caret is not underlined. The contextual menu offers up to six guesses, "Ignore Spelling" and "Learn Spelling"; a correction is applied with replace(_:with:) and therefore honours change tracking. The preference is AppSettings.checkSpelling.

Grammar checking is not requested, and there is no autocorrection or text replacement.

# 14. Accessibility

The canvas is one accessibility element with role textArea, label "Document" and identifier document-canvas. It exposes the story being edited as flat text: paragraphs in reading order (table cells included) separated by newlines, inline objects replaced by a space (flatText()). On top of that it implements the value, number of characters, selected text, selected range (get and set), string for range, frame for range, and line queries, and posts valueChanged / selectedTextChanged notifications. accessibilityHelp gives "Page x of y".

Limits: a "line" is a paragraph, not a visual line; the visible range is reported as the whole story; no attributed text (styles, headings, links, tracked changes) and no child elements for tables, images or equations are exposed; the value cannot be set through accessibility; the flat text is rebuilt on each query.

# 15. Limits

  • Cells hold paragraphs only; there are no nested tables (setContainer and insertFragment flatten them). Tables and breaks exist in the body only.
  • Bidirectional text: the caret is at the primary offset reported by CTLineGetOffsetForStringIndex, there is no split caret, and Left/Right move in logical order. A selection rectangle on a line spans from the smaller to the larger of the two caret x positions (ParagraphLayout.rects(for:lines:)), which is not the exact set of selected glyphs when a range mixes directions.
  • Not provided by the custom view (present in NSTextView, absent from the code): Writing Tools, autocorrection, text replacement, inline predictions, grammar checking, Look Up / data detectors, the Services menu for the selection (no validRequestor(forSendType:returnType:)), the system find bar (NSTextFinder; the app has its own find panel), the Touch Bar, dragging text, pinch-to-zoom, and the optional NSTextInputClient members.
  • The file-level claim in EditorCanvasView+Keyboard.swift that dictation works through the text input client is not exercised by any test in the repository; the self-test covers direct typing, key bindings, mouse and pasteboard only.
  • Text committed through unmarkText() rather than insertText was composed with tracking off and is kept as is, so it carries no insertion revision.
  • characterIndex(for:) returns an offset in the paragraph under the pointer, whereas the other members speak about the caret's paragraph.
Files main
Menu

Theme