SPB Git

Repository Public License

testing.md 8.0 KB

# Testing

What is tested, where it runs, what the last run produced, and what is not covered. Results below are from 2026-11-20 on the final code of that day.

# Summary

Suite Command Environment Result
Package tests Scripts/build.sh kit Mac host (Apple M5 Max, macOS 27.0), Swift Testing 79 passed, 0 failed, 2 skipped (opt-in benchmarks)
UI tests, iPhone Scripts/build.sh uitest "ORIX iPhone" = iPhone 17 Pro simulator, iOS 26.5 13 passed, 0 failed
UI tests, iPad ORIX_SIM="ORIX iPad" Scripts/build.sh uitest "ORIX iPad" = iPad Pro 13-inch (M5) simulator, iOS 26.5, landscape 13 passed, 0 failed (see the note below)
Benchmarks Scripts/benchmark.sh Mac host 2 passed (results in benchmarks/)
Physical device — — Not run: no iPhone or iPad was connected
Code coverage — — Not measured

iPad note: three full runs were made on the final code. Two passed 13 of 13. In one, testImportScannedPDFRecognizeAndSearch failed before touching the app because the test bundle did not return its fixture file; the same test then passed twice when run alone, and the fixture is present in the built bundle. It is treated as a test-harness flake, not an app defect, and it was not explained.

swift test reports 81 tests: 79 run and pass, and the two benchmark tests are skipped unless ORIX_BENCH is set.

# Why most tests run on the Mac

ORIXKit has no UI and builds for macOS as well as iOS. Vision, PDFKit, Core Image, Core Text and SwiftData are the same frameworks on both, so the engine tests exercise the real recognizer and the real PDF reader in a few seconds, without a simulator. They do not prove anything about iPhone hardware, memory limits or the camera.

# Package tests (Packages/ORIXKit/Tests/ORIXKitTests)

Test pages are drawn with Core Text into bitmaps (Support.swift), so every page has an exact ground truth.

File Suites What they check
EngineTests.swift Geometry, TextMetrics, ReadingOrder, TableExtractor, Quality, WorkflowModel Quad validity and rotation, stable cache keys, CER / WER arithmetic, two-column and title ordering, table grid reconstruction with empty cells, CSV escaping, each quality heuristic on a page built to trigger it, rename tokens, preset stability, JSON round trip, safe file names with Unicode
PipelineTests.swift Recognition English and French accuracy thresholds, Fast mode, languages read from Vision, word boxes inside their line, skew measurement, cancellation, a table through real recognition
SearchablePDF Text extraction, findString, search-hit bounds against the recognized word box, accents and metadata, page order and page sizes (A4, Letter landscape, receipt proportions), fixed paper size, password (locked, wrong, right), no text layer when not asked, empty document refused
Store Drafts invisible until commit and purged after a simulated crash, relaunch persistence with a Unicode title, bad image and empty draft refused, corrupted manifest reported, page duplicate / delete with shared originals, retake, merge / split / extract, PDF import including protected and corrupted files and an unusual page size, low-storage guard
Processor Scan → recognize → export → search end to end, text discarded when geometry changes, sideways page recognized after rotation, straightening a tilted page, each compression preset re-read by OCR and size ordering, imported text pages passed through, image-only PDF pages recognized and recompressed, text / CSV / image export, filters lift grey paper to white, Black & White keeps faint text, perspective crop, preview cache keyed by edit, export cancellation leaves no file
WorkflowEngine Preset end to end with monotonic progress, rename and folder result, "no table found" outcome, every preset on a real document, failing step stops the run, cancellation
EnhancementQuality No grey halo around text after background cleanup
LibraryTests.swift Library Index refresh with accent-insensitive text search, sorting and filtering, unique titles, trash / restore / permanent delete, rebuild from files and 30-day purge, folders, rename, duplicate, favourites, workflow persistence, on-disk index reopening
BenchmarkTests.swift Benchmarks OCR accuracy on the corpus, timings (opt-in)

# UI tests (UITests/ORIXScanUITests.swift)

The app is launched with -ORIXUITest YES (a store under tmp, never the user's library), -ORIXReset YES and, for most tests, -ORIXDemo YES, which makes the debug build draw five sample documents and process them with the real engines.

Test Flow
testFirstLaunchIsUsefulImmediately No alert, no onboarding; Scan and both imports on screen; scanning without a camera explains itself; empty library offers scan and import
testLibrarySearchDeleteAndRestore Five documents listed; batch select → delete → Recently Deleted → restore; search by a word that exists only in recognized text
testTextSearchReviewTablesAndSearchablePDF Text search with match count; review mode; a table extracted from the invoice; all three export sizes measured; search inside the exported PDF gives "1 of 1", then "No matches"
testPageEditorFiltersCompareAndCrop Filter selection, before / after, crop editor
testOrganizePagesAndSplit Three pages; split creates two new documents
testRunActionOnDocumentAndBuildOne Presets listed; new action opens the builder; Extract Text runs on a document and produces a file
testImportScannedPDFRecognizeAndSearch An image-only PDF goes through the real import code, automatic recognition, text search and search inside the exported PDF
testDocumentsPersistAfterRelaunch Terminate, relaunch without reset: documents and recognized text are still there
testDarkMode Home and Library in dark appearance
testLargestDynamicType Home and Document usable at the AX XXL text size
testFrenchLocalization Home and Library in French (Canada)
testSettingsListsVisionLanguages The language list contains what Vision reports (en-US and fr-FR asserted)
testColdLaunchTime XCTApplicationLaunchMetric, five launches

Screenshots attached by these tests are the ones in docs/screenshots/ (Scripts/shots.py collects them).

# OCR benchmark

Scripts/benchmark.sh recognizes 17 synthetic pages in the three modes and writes benchmarks/ocr-accuracy.json. Categories: English, French, mixed, receipt, low resolution, skew, multi-column, difficult lighting, blur, and two script fonts standing in for handwriting. The README carries the table.

Limits of this benchmark, stated plainly:

  • The pages are rendered, not photographed. Paper texture, lens blur, curvature, moiré and real shadows are absent.
  • The corpus is small (5 to 8 lines per page).
  • "Handwriting-like" rows are computer fonts.
  • It ran on macOS; the Vision revision on a given iOS version can differ.

# Not covered

Area Why
Document camera (edge detection, flash, multi-page, retake) Needs a camera; system UI
Photo picker and Files picker screens System UI running outside the app; the import code behind them is covered
Share sheet, printing, Save to Files destinations System UI; the files handed to them are checked
App Lock with Face ID / Touch ID Needs a device
VoiceOver Not run
Low storage and memory pressure on a device Guard code is unit-tested; real conditions were not produced
Large documents (hundreds of pages) Largest measured: 25 pages
Interruption by a phone call, backgrounding mid-recognition Not exercised
Shortcuts intents run from the Shortcuts app Their pipeline is the tested engine code; the intents themselves were not invoked

# Manual device pass (to do before TestFlight)

The list is in ../release/release-checklist.md, section 2.

Files main
Menu

Theme