Repository Public License
1<!--2ORIX Scan3File: docs/architecture/ocr-engine.md4Author: Simon-Pierre Boucher5Contact: contact@spboucher.ai6-->78# OCR engine910Text recognition in ORIX Scan is Apple's Vision framework, called on the device. The code is in11`Packages/ORIXKit/Sources/ORIXKit/OCR/`.1213## Local by construction1415`OCREngine` creates a `VNRecognizeTextRequest` and performs it with a `VNImageRequestHandler` on a `CGImage` held in memory.16The repository contains no networking code: there is no `URLSession`, no socket and no third-party SDK in `ORIXKit`,17`ORIXDesign` or the app target, and the package has no dependencies. Recognized text is written to a JSON file next to the18page (`ocr/<page>.json`) and to the on-device SwiftData and Spotlight indexes. It leaves the device only when the user19shares an export.2021## Pipeline2223```mermaid24flowchart TD25 Page["PageRecord + PageEdit"] --> Render["DocumentProcessor.renderedImage<br/>(crop, straighten, rotate, filter; max 4200 px)"]26 Render --> BW{"filter is blackWhite?"}27 BW -- yes --> Grey["recognize the greyscale render instead"]28 BW -- no --> Req29 Grey --> Req["VNRecognizeTextRequest"]30 Req --> Obs["observations: text, confidence, line box"]31 Obs --> Words["word boxes via boundingBox(for:)"]32 Words --> Order["ReadingOrder.sorted (XY-cut)"]33 Order --> Result["OCRPage with geometryKey"]34 Result --> Save[("ocr/<page>.json")]35 Save --> Text["plainText, search index"]36 Save --> Tables["TableExtractor"]37 Save --> Layer["PDF text layer"]38```3940## What `OCREngine` configures4142`OCREngine.recognize(_:options:geometryKey:progress:)` takes an upright `CGImage` and `OCROptions`:4344| Option | Effect on the request |45|---|---|46| `mode: .fast` | `recognitionLevel = .fast` |47| `mode: .accurate` | `recognitionLevel = .accurate` |48| `mode: .automatic` | `recognitionLevel = .accurate` and `automaticallyDetectsLanguage = true` |49| `languages` | Filtered against `supportedRecognitionLanguages()` of the request, then set as `recognitionLanguages` when not empty. With no usable language, automatic language detection is turned on |50| `usesLanguageCorrection` | Passed through |5152**Languages are never hard-coded.** `OCREngine.supportedLanguages(mode:)` returns what the installed Vision revision53reports for the level of that mode. The Settings language list (`LanguagePicker`) is built from that call, so the app cannot54advertise a language the system does not support.5556**Cancellation.** `recognize` checks for cancellation first, then wraps the work in `withTaskCancellationHandler`. A small57lock-protected box gives the handler the in-flight `VNRequest` so it can call `cancel()`. A cancelled run throws58`ORIXError.cancelled`.5960**Progress.** When a progress closure is supplied, the request's `progressHandler` forwards Vision's fraction.61`DocumentProcessor.recognize` scales it across the pages of the document.6263## `OCRPage`6465| Field | Meaning |66|---|---|67| `imageWidth`, `imageHeight` | Pixel size of the image that was recognized |68| `lines` | `[OCRLine]` in reading order |69| `mode`, `languages` | What was requested |70| `geometryKey` | Geometry of the edit the boxes belong to |71| `recognizedAt`, `duration` | Time stamp and seconds spent |7273An `OCRLine` carries `text`, `recognizedText`, `confidence` (Vision's value for the top candidate), a `Quad`, and `words`.74Each `OCRWord` is a whitespace-separated token (`OCREngine.tokenRanges`, punctuation stays attached) with the box Vision75returns for that range. All coordinates are converted from Vision's bottom-left origin to the top-left, normalized space of76`Quad`.7778## Reading order7980`ReadingOrder` is a recursive XY-cut over line bounding boxes. At each level it first tries to split into columns along81vertical whitespace bands at least `max(1.8 × median line height, 0.025)` wide, then into rows along horizontal bands of82`1.4 ×` the median line height, to a depth of 12. A column split at the top level is accepted only when every group has at83least two lines, which keeps a short stray line from creating a false column. Leaves are sorted top to bottom, and left to84right within a visual row. `OCRPage.plainText` joins the lines of a block with newlines and blocks with a blank line.8586## Skew measurement8788`OCRPage.skewDegrees` is the median baseline angle of lines with at least six characters. For a line with three or more89words the angle is the least-squares slope of the word-box centers, corrected for the image aspect ratio; shorter lines fall90back to the line box.9192The word-center method exists because of a measured behaviour: on a page rotated by 4°, Vision's accurate level returned93rotated line quadrilaterals, but the fast level returned axis-aligned line boxes, giving an angle of 0. The word boxes follow94the tilted baseline at both levels. `DocumentProcessor.deskew` uses a fast probe at 1600 px and applies a correction only95between 0.3° and 15°.9697## Table extraction9899`TableExtractor.extract(from:)` works from geometry only:1001011. **Tokens**: the word boxes of each line (or the line box when the line was edited or has no words).1022. **Rows**: tokens whose vertical centers are within 0.6 of the smaller height are grouped.1033. **Segments**: in a row, tokens closer than about 1.25 line heights are merged into one cell candidate.1044. **Candidates**: runs of at least two consecutive rows holding two or more segments, with no vertical gap larger than three105 line heights.1065. **Columns**: an occupancy histogram along x; gutters are intervals no segment crosses (a tolerance of one tenth of the107 rows applies from eight rows up). Segments are assigned to the column containing their center.1086. **Uncertainty**: a cell is flagged when its segment spans a gutter, when two segments landed in the same cell, or when109 recognition confidence is below 0.5. `confidence` is the share of rows that matched the grid exactly multiplied by the110 mean recognition confidence.111112`ExtractedTable.csv` is RFC 4180 (CRLF, quoting when a field contains a comma, a quote or a line break).113114Limits, stated in the interface as well:115116- Merged cells, nested headers and cell text wrapped on several lines are not reconstructed faithfully.117- Ruling lines are ignored; only text positions are used.118- **Measured**: in a synthetic table, Vision did not return cells made of a single isolated character (a lone digit such as119 `3` or `7`), at any level and with or without language correction. Those cells come out empty. Two-character cells in the120 same table were recognized.121- The Tables screen lets the user correct any cell before exporting; those corrections are applied to that export only.122123## Text correction124125The Review screen (`App/Sources/Document/DocumentTextView.swift`) edits `OCRLine.text`; `recognizedText` keeps Vision's126output so a correction can be reverted (`isEdited`). Corrections are saved to the page's JSON and affect copied text, the127text export, the search index and the PDF text layer. An edited line is drawn in the PDF as one run across the line box,128because its word boxes no longer match its words.129130## Geometry key131132Recognized boxes are valid only for the geometry they were measured on. `PageEdit.geometryKey` hashes the crop quad, the133quarter turns and the straightening angle, and ignores colour settings. `DocumentProcessor.updateEdits` deletes a page's134recognition when the geometry changes, and every consumer (`text(of:in:)`, `tables(of:)`, `exportPDF`) compares135`OCRPage.geometryKey` with the page's current key, so a stale text layer cannot be exported.136137Imported PDF pages that already contain text are not recognized again; their text is read with PDFKit138(`PDFTools.embeddedText`).139140## Handwriting141142ORIX Scan does not offer a handwriting feature. Vision's accurate level may read clear handwriting, but this has not been143validated on real handwritten documents, so the app makes no claim about it.144