Repository Public
1<!--2ORIX Docs3File: docs/development/testing.md4Description: The engine test targets, the interaction self-test, the recovery script and the XCUITest suite: what each covers and how to run it.5Author: Simon-Pierre Boucher6Contact: contact@spboucher.ai7-->8# Testing910There are four kinds of automated check:1112| Check | Where | Runs |13|---|---|---|14| Engine tests | `Packages/ORIXDocsKit/Tests/` (Swift Testing) | `swift test --no-parallel`, no window |15| Interaction self-test | `App/Sources/App/SelfTest.swift` | inside the built app, `-ORIXSelfTest <report.json>` |16| Recovery test | `Scripts/recovery-test.sh` | three launches of the built app |17| XCUITest suite | `UITests/EditorUITests.swift` | Xcode UI testing — **not run**, see below |1819`Scripts/test.sh [engines|app|all]` runs the first three and writes `docs/reports/test-results.json` (date, number of test targets, tests and suites with the overall result, self-test scenario counts, last line of the recovery script). That file is the record of the last run; this document does not restate its numbers as a status.2021## Engine tests2223### How to run2425```sh26cd Packages/ORIXDocsKit27swift test --no-parallel # everything28swift test --no-parallel --filter KeepRuleTests29ORIX_ONLY=DocModel,DocModelTests swift test # only the listed targets are part of the package30```3132**Why serial.** Swift Testing runs tests in parallel by default. Tests that lay out documents from many threads at once can stall in the system font registry (the reason `FontCache` reads the installed-family list once, see [layout-engine.md](../architecture/layout-engine.md#limits)), so the suite is run with `--no-parallel`, as `Scripts/test.sh` does.3334Environment variables read by the tests:3536| Variable | Effect |37|---|---|38| `ORIX_ONLY` | Comma-separated target names; `Package.swift` drops every other target. Lets one engine be tested while another does not compile. |39| `ORIX_SLOW_TESTS=1` | Enables the LibreOffice conversions in `DocDOCXTests` (`ExternalValidationTests`, the compatibility report). Without it they print `ORIX-DOCX-SKIP: …` and return. |40| `ORIX_ARTIFACTS=<dir>` | `TemplateRenderingTests` writes each template's PDF and PNG renderings of its first three pages there. |41| `ORIX_DOCX_REPORT_DIR=<dir>` | Where `CompatibilityReportTests` writes the DOCX compatibility report (set by `Scripts/docx-compat.sh`). |42| `ORIX_AI_TESTS=1` | Enables `OnDeviceModelTests` in `DocAutomationTests`, which need the system's on-device model. |43| `ORIX_AUTOMATION_HOME=<dir>` | Moves the automation history and workflow folder; the tests point it at a temporary directory. |4445Tests that depend on an external tool (Poppler, pandoc, LibreOffice, `textutil`) return early with a printed `…-SKIP` line when the tool is missing. A skipped check still counts as a passed test in the totals.4647### Targets4849The counts are **source counts**: the number of `@Test` attributes and `@Suite` attributes in each target's files (`grep -cE '^\s*@Test'`), taken when this document was written. A parameterised test counts once here, whatever its number of arguments.5051| Target | `@Test` | `@Suite` | Covers |52|---|---|---|---|53| `DocModelTests` | 74 | 9 | Typing and typing attributes; composed-character safety (`UnicodeTests`); paragraph splitting and joining; character and paragraph formatting, style inheritance; lists and numbering, captions, cross-references; copy/paste fragments and JSON round trip; track changes including a randomised accept/reject invariant; comments, find and replace; table editing, sorting and cell formulas. |54| `DocContainerTests` | 5 | 1 | The ZIP reader and writer (`ZipTests`). |55| `DocDOCXTests` | 73 | 9 | DOCX import of a pandoc/LibreOffice corpus and of hand-authored packages; export → reimport round trips; export checked by independent tools; hostile input; scale; LaTeX ↔ OMML math conversion; the compatibility report. See [docx.md](../architecture/docx.md). |56| `DocFormatsTests` | 110 | 13 | Native container ([persistence.md](../architecture/persistence.md)), Markdown, HTML and RTF import and export, plain text, PDF text import, the `DocumentIO` facade, Unicode torture cases, hostile input, cross-checks against pandoc and `textutil`. |57| `DocLayoutTests` | 60 | 11 | Page geometry, breaks and sections, keep rules, footnotes, table layout, generated content, typesetting, geometry queries ([pagination.md](../architecture/pagination.md)); PDF publishing ([pdf.md](../architecture/pdf.md)); math robustness (hostile and random LaTeX never crashes); every template lays out and publishes. |58| `DocReferencesTests` | 56 | 10 | BibTeX, RIS and CSL-JSON parsing; the citation engine; APA, Chicago, IEEE, MLA and Vancouver formatting. See [references.md](../architecture/references.md). |59| `DocReviewTests` | 48 | 9 | Document comparison: content, formatting, moves, tables, inline objects and stories, Unicode, invariants, reports, performance. See [review.md](../architecture/review.md). |60| `DocAuditTests` | 49 | 9 | Auditor checks (headings, references, formatting and review, pages, language), behaviour, search, acronyms, research tools. See [audit.md](../architecture/audit.md). |61| `DocTemplatesTests` | 43 | 6 | Template catalogue, integrity, features, context, language, genre. |62| `DocAutomationTests` | 64 | 8 | The `orixdocs` command line, workflow steps and runner, text-intelligence support and runner, the on-device-only guarantee, confinement of images referenced by imported documents. See [automation.md](automation.md). |63| **Total** | **581** | **84** | |6465There are no unit tests in the app target; the app's behaviour is covered by the self-test below.6667## Interaction self-test6869`SelfTest` is compiled into the application and started by the launch option `-ORIXSelfTest <report.json>` ([setup.md](setup.md#launch-options)). It synthesises `NSEvent` key and mouse events and sends them to the document window (`window.sendEvent`, and `NSApp.mainMenu.performKeyEquivalent` for ⌘ shortcuts), so each scenario goes through the same path as a person at the keyboard: window → canvas → input context → `NSTextInputClient` / `doCommand(by:)` → editor → layout. After each step it compares the document state. Some steps call `DocumentSession.run` or `perform` directly (inserting a table, toggling tracking); they are not key presses.7071It writes a JSON report (`scenarios` with name, `passed`, `failures`, `milliseconds`; totals; date; system version) and the process exits.7273```sh74Scripts/build.sh75"build/DerivedData/Build/Products/Debug/ORIX Docs.app/Contents/MacOS/ORIX Docs" \76 -ApplePersistenceIgnoreState YES -ORIXTemplate blank \77 -ORIXSelfTest "$PWD/build/selftest.json" -checkSpelling NO -AppleLanguages "(en)"78```7980The 19 scenarios, in order:8182| # | Scenario | Checks |83|---|---|---|84| 1 | typing, Return and Backspace | Two typed paragraphs; five Backspaces; caret position. |85| 2 | Unicode typing deletes whole characters | A ZWJ family emoji and CJK text; Backspace removes one composed character at a time. |86| 3 | arrow keys, word and line movement | ⌥←, ⌥⌫, ⌘←, ⌘→, ⇧← selection replaced by typing, forward delete. |87| 4 | vertical movement keeps the column | ↓ and ↑ through a short line restore the column. |88| 5 | undo, redo and select all | ⌘A + Delete, undo, redo, a second undo. |89| 6 | typing is coalesced into one undo step per burst | One undo removes a typed word. |90| 7 | bold shortcut applies to typed text | ⌘B with a caret formats only what is typed next. |91| 8 | smart quotes and dashes | `"Hi" -- it's` becomes `“Hi” — it’s`. |92| 9 | mouse: click, shift-click, double and triple click, drag | Caret placement, extension, word and paragraph selection, drag across paragraphs, typing over the selection. |93| 10 | copy, cut and paste through the pasteboard | ⌘C/⌘V keep bold; ⌘X then ⌘V restores; plain text and RTF flavours are present. |94| 11 | lists: Tab indents, Return on an empty item leaves the list | Labels "1." and "a."; leaving the list. |95| 12 | table: insert, type, Tab between cells, new row at the end | Cell contents, added row, ⇧Tab, Delete Rows. |96| 13 | footnote: insert, type in the note, Escape returns | Note text, return to body, deleting the mark deletes the note. |97| 14 | header: edit, page field, return to body | Header text and page field stored and laid out. |98| 15 | track changes: insert, delete, accept and reject | Two tracked changes, Reject All, undo, Accept All. |99| 16 | comment anchored on the selection | A comment anchored on the double-clicked word. |100| 17 | find and replace | Two matches, Find Next, Replace All. |101| 18 | page breaks create pages; many pages scroll with the caret | Four pages; the caret's page is visible. |102| 19 | save, reopen and autosave through NSDocument | Save to a file and read it back; edit and `autosave`; an external overwrite stays on disk ([persistence.md](../architecture/persistence.md#external-modification)). |103104Limits of the self-test: events are synthesised inside the process, so it does not prove that the window server delivers events, nor exercise an input method, dictation, VoiceOver or drag and drop from another application. The run sets `UndoManager.groupsByEvent = false`, which is why `DocumentSession.registerUndo` opens explicit undo groups ([text-engine.md](../architecture/text-engine.md#7-undo)).105106## Recovery test107108```sh109Scripts/build.sh110Scripts/recovery-test.sh # ORIX_APP=<path to .app> and ORIX_RECOVERY_WAIT=<seconds> are optional111```112113Creates a saved document through the app, types into it with `-ORIXType` and a two-second autosave delay, sends `SIGKILL`, reopens the file and compares the text; prints `PASS` or `FAIL`. What it does and does not prove is set out in [persistence.md](../architecture/persistence.md#recovery-after-a-kill).114115## XCUITest suite116117`UITests/EditorUITests.swift` holds ten XCUITest methods that drive the application through the accessibility interface (the canvas is found by its identifier `document-canvas` and its text read from its accessibility value): typing and Backspace, Unicode typing, undo/redo and Select All, arrow keys and word deletion, the Bold shortcut and toolbar state, modes, inserting a table from the menu, footnotes and track changes, save and reopen, and autosave surviving a killed process. The target is `ORIXDocsUITests` in `project.yml`, attached to the `ORIXDocs` scheme.118119**This suite has not been run.** XCUITest needs macOS Automation Mode (UI automation permission) to be authorised on the machine by its owner, and that authorisation was not given. No result is claimed for it. The interaction self-test was written to cover the same ground without that permission, and it is the check that was used instead; the two overlap but are not equivalent — the XCUITest suite goes through the real accessibility and event system, the self-test does not.120121To run it once Automation Mode is authorised:122123```sh124xcodegen generate125xcodebuild -project ORIXDocs.xcodeproj -scheme ORIXDocs -derivedDataPath build/DerivedData test126```127128## Other reports129130* `Scripts/docx-compat.sh` runs the DOCX corpus cycle and writes `docs/reports/docx-compatibility.md` and `.json`.131* `swift run -c release orixbench` writes the benchmark figures ([performance.md](performance.md)).132