Repository Public License
1<!--2ORIX Scan3File: docs/development/testing.md4Author: Simon-Pierre Boucher5Contact: contact@spboucher.ai6-->78# Testing910What is tested, where it runs, what the last run produced, and what is not covered. Results below are from 2026-11-20 on the11final code of that day.1213## Summary1415| Suite | Command | Environment | Result |16|---|---|---|---|17| Package tests | `Scripts/build.sh kit` | Mac host (Apple M5 Max, macOS 27.0), Swift Testing | 79 passed, 0 failed, 2 skipped (opt-in benchmarks) |18| UI tests, iPhone | `Scripts/build.sh uitest` | "ORIX iPhone" = iPhone 17 Pro simulator, iOS 26.5 | 13 passed, 0 failed |19| UI tests, iPad | `ORIX_SIM="ORIX iPad" Scripts/build.sh uitest` | "ORIX iPad" = iPad Pro 13-inch (M5) simulator, iOS 26.5, landscape | 13 passed, 0 failed (see the note below) |20| Benchmarks | `Scripts/benchmark.sh` | Mac host | 2 passed (results in `benchmarks/`) |21| Physical device | — | — | Not run: no iPhone or iPad was connected |22| Code coverage | — | — | Not measured |2324iPad note: three full runs were made on the final code. Two passed 13 of 13. In one, `testImportScannedPDFRecognizeAndSearch`25failed before touching the app because the test bundle did not return its fixture file; the same test then passed twice when26run alone, and the fixture is present in the built bundle. It is treated as a test-harness flake, not an app defect, and it27was not explained.2829`swift test` reports 81 tests: 79 run and pass, and the two benchmark tests are skipped unless `ORIX_BENCH` is set.3031## Why most tests run on the Mac3233`ORIXKit` has no UI and builds for macOS as well as iOS. Vision, PDFKit, Core Image, Core Text and SwiftData are the same34frameworks on both, so the engine tests exercise the real recognizer and the real PDF reader in a few seconds, without a35simulator. They do not prove anything about iPhone hardware, memory limits or the camera.3637## Package tests (`Packages/ORIXKit/Tests/ORIXKitTests`)3839Test pages are drawn with Core Text into bitmaps (`Support.swift`), so every page has an exact ground truth.4041| File | Suites | What they check |42|---|---|---|43| `EngineTests.swift` | Geometry, TextMetrics, ReadingOrder, TableExtractor, Quality, WorkflowModel | Quad validity and rotation, stable cache keys, CER / WER arithmetic, two-column and title ordering, table grid reconstruction with empty cells, CSV escaping, each quality heuristic on a page built to trigger it, rename tokens, preset stability, JSON round trip, safe file names with Unicode |44| `PipelineTests.swift` | Recognition | English and French accuracy thresholds, Fast mode, languages read from Vision, word boxes inside their line, skew measurement, cancellation, a table through real recognition |45| | SearchablePDF | Text extraction, `findString`, **search-hit bounds against the recognized word box**, accents and metadata, page order and page sizes (A4, Letter landscape, receipt proportions), fixed paper size, password (locked, wrong, right), no text layer when not asked, empty document refused |46| | Store | Drafts invisible until commit and purged after a simulated crash, relaunch persistence with a Unicode title, bad image and empty draft refused, corrupted manifest reported, page duplicate / delete with shared originals, retake, merge / split / extract, PDF import including protected and corrupted files and an unusual page size, low-storage guard |47| | Processor | Scan → recognize → export → search end to end, text discarded when geometry changes, sideways page recognized after rotation, straightening a tilted page, **each compression preset re-read by OCR** and size ordering, imported text pages passed through, image-only PDF pages recognized and recompressed, text / CSV / image export, filters lift grey paper to white, Black & White keeps faint text, perspective crop, preview cache keyed by edit, export cancellation leaves no file |48| | WorkflowEngine | Preset end to end with monotonic progress, rename and folder result, "no table found" outcome, every preset on a real document, failing step stops the run, cancellation |49| | EnhancementQuality | No grey halo around text after background cleanup |50| `LibraryTests.swift` | Library | Index refresh with accent-insensitive text search, sorting and filtering, unique titles, trash / restore / permanent delete, rebuild from files and 30-day purge, folders, rename, duplicate, favourites, workflow persistence, on-disk index reopening |51| `BenchmarkTests.swift` | Benchmarks | OCR accuracy on the corpus, timings (opt-in) |5253## UI tests (`UITests/ORIXScanUITests.swift`)5455The app is launched with `-ORIXUITest YES` (a store under `tmp`, never the user's library), `-ORIXReset YES` and, for most56tests, `-ORIXDemo YES`, which makes the debug build draw five sample documents and process them with the real engines.5758| Test | Flow |59|---|---|60| `testFirstLaunchIsUsefulImmediately` | No alert, no onboarding; Scan and both imports on screen; scanning without a camera explains itself; empty library offers scan and import |61| `testLibrarySearchDeleteAndRestore` | Five documents listed; batch select → delete → Recently Deleted → restore; search by a word that exists only in recognized text |62| `testTextSearchReviewTablesAndSearchablePDF` | Text search with match count; review mode; a table extracted from the invoice; all three export sizes measured; search inside the exported PDF gives "1 of 1", then "No matches" |63| `testPageEditorFiltersCompareAndCrop` | Filter selection, before / after, crop editor |64| `testOrganizePagesAndSplit` | Three pages; split creates two new documents |65| `testRunActionOnDocumentAndBuildOne` | Presets listed; new action opens the builder; Extract Text runs on a document and produces a file |66| `testImportScannedPDFRecognizeAndSearch` | An image-only PDF goes through the real import code, automatic recognition, text search and search inside the exported PDF |67| `testDocumentsPersistAfterRelaunch` | Terminate, relaunch without reset: documents and recognized text are still there |68| `testDarkMode` | Home and Library in dark appearance |69| `testLargestDynamicType` | Home and Document usable at the AX XXL text size |70| `testFrenchLocalization` | Home and Library in French (Canada) |71| `testSettingsListsVisionLanguages` | The language list contains what Vision reports (en-US and fr-FR asserted) |72| `testColdLaunchTime` | `XCTApplicationLaunchMetric`, five launches |7374Screenshots attached by these tests are the ones in `docs/screenshots/` (`Scripts/shots.py` collects them).7576## OCR benchmark7778`Scripts/benchmark.sh` recognizes 17 synthetic pages in the three modes and writes79[`benchmarks/ocr-accuracy.json`](benchmarks/ocr-accuracy.json). Categories: English, French, mixed, receipt, low resolution,80skew, multi-column, difficult lighting, blur, and two script fonts standing in for handwriting. The README carries the table.8182Limits of this benchmark, stated plainly:8384- The pages are rendered, not photographed. Paper texture, lens blur, curvature, moiré and real shadows are absent.85- The corpus is small (5 to 8 lines per page).86- "Handwriting-like" rows are computer fonts.87- It ran on macOS; the Vision revision on a given iOS version can differ.8889## Not covered9091| Area | Why |92|---|---|93| Document camera (edge detection, flash, multi-page, retake) | Needs a camera; system UI |94| Photo picker and Files picker screens | System UI running outside the app; the import code behind them is covered |95| Share sheet, printing, Save to Files destinations | System UI; the files handed to them are checked |96| App Lock with Face ID / Touch ID | Needs a device |97| VoiceOver | Not run |98| Low storage and memory pressure on a device | Guard code is unit-tested; real conditions were not produced |99| Large documents (hundreds of pages) | Largest measured: 25 pages |100| Interruption by a phone call, backgrounding mid-recognition | Not exercised |101| Shortcuts intents run from the Shortcuts app | Their pipeline is the tested engine code; the intents themselves were not invoked |102103## Manual device pass (to do before TestFlight)104105The list is in [`../release/release-checklist.md`](../release/release-checklist.md), section 2.106