SPB Git

Repository Public License

core-api.md 23.2 KB 369 lines markdown

1# ORIX Convert — core API (`ConvertCore`)23Everything the app needs between "the user dropped files" and "the converted files are in place" lives in4`Packages/ORIXConvertCore` (module `ConvertCore`). The core contains **no conversion code**: engines5(`Packages/Engine*`) implement `ConversionEngine`; the core detects, plans, schedules, validates, writes and records.67All public types are `Sendable`. Stateful services are actors; everything else is a value type.8Every sample below is compiled and executed by `Tests/ConvertCoreTests/DocSamplesTests.swift`.910## Map of the module1112| Area | Types | File |13|---|---|---|14| Contract (shared, do not change) | `FormatID`, `FileFormat`, `FormatCatalog`, `ConversionEngine`, `ConversionPath`, `ConversionRequest`, `ConversionContext`, `ConversionOutput`, `FileInspection`, `ValidationCheck`, `ConversionError`, `BundledTools`, `ProcessRunner`… | `Formats.swift`, `Engine.swift`, `Tools.swift` |15| Detection | `FileTypeDetector`, `DetectionResult`, `TextEncoding`, `FormatEquivalence` | `Detection/` |16| Registry | `FormatRegistry`, `RegistryCounts`, `MatrixRow`, `UnavailableEngine` | `FormatRegistry.swift` |17| Capabilities and planning | `ConversionCapabilityResolver`, `ConversionCapabilities`, `ConversionTarget`, `ConversionPlanner`, `ConversionPlan` | `Planner.swift` |18| Safe output | `WorkDirectoryManager`, `OutputValidator`, `SafeWriter`, `ConflictPolicy`, `CommittedOutput`, `OutputNameTemplate`, `OutputNameContext`, `AppPaths` | `SafeOutput.swift` |19| Queue | `ConversionJobManager`, `ConversionQueue`, `ConversionJobRequest`, `ConversionJob`, `JobState`, `JobPriority`, `OutputDestination`, `JobProgress`, `JobLogLine`, `DestinationConflict`, `QueueEvent`, `QueueSnapshot`, `QueueStatus`, `QueueConfiguration`, `ConcurrencyLimits`, `ResourceState`, `ThermalLevel`, `ResourceMonitoring`, `SystemResourceMonitor`, `ETAEstimator` | `Jobs/` |20| History | `ConversionHistoryStore`, `HistoryEntry`, `HistoryQuery`, `WorkflowRunRecord`, `HistoryError` | `History.swift` |21| Presets | `PresetStore`, `ConversionPreset`, `PresetError` | `Presets.swift` |22| Workflows | `Workflow`, `WorkflowNode`, `WorkflowCondition`, `Comparison`, `Glob`, `WorkflowErrorPolicy`, `WorkflowValidator`, `WorkflowValidation`, `WorkflowIssue`, `WorkflowRunner`, `WorkflowRunOptions`, `WorkflowRunResult`, `WorkflowItemResult`, `WorkflowLibrary`, `WorkflowError` | `Workflows.swift` |23| Watched folders | `FolderWatcher`, `WatchRule`, `WatchActivity`, `WatchEvent`, `WatcherConfiguration`, `WatchError` | `FolderWatcher.swift` |24| Inspection | `FileInspector`, `InspectedFile`, `FileFacts`, `SmartDropAnalyzer`, `SmartDropAnalysis`, `SmartDropItem`, `SmartDropGroup`, `SmartDropFolder` | `Inspection.swift` |2526## Setting up2728```swift29let history = try ConversionHistoryStore(url: supportFolder.appendingPathComponent("history.sqlite"))30let manager = ConversionJobManager(engines: engines, history: history,31                                   workDirectories: WorkDirectoryManager(root: temp.file("work")))32let registry = manager.registry33let queue = manager.queue34```3536In the app, `engines` is `EngineImage.makeEngines() + EngineMedia.makeEngines() + …`, and the defaults place37everything where it belongs: `ConversionHistoryStore()` opens38`~/Library/Application Support/ORIX Convert/history.sqlite`, `WorkDirectoryManager()` uses39`$TMPDIR/ai.spboucher.orix.convert/work` (mode 0700; directories left by dead processes are swept when the queue is40created). `AppPaths` lists the locations; `ORIX_CONVERT_SUPPORT_DIR` overrides the support folder for tools and tests.4142The registry is a snapshot: engine availability (`availability()`) is read once, when it is built. Paths of an43unavailable engine are excluded from `registry.paths` and listed in `registry.unavailableEngines` with the reason.4445Put the heavy engines' identifiers in `QueueConfiguration.heavyEngineIDs` (the core cannot guess them):4647```swift48QueueConfiguration(maxConcurrentJobs: 6, maxHeavyJobs: 1, heavyEngineIDs: ["documents-office", "media"], reduceOnBattery: true)49```5051## Detecting and inspecting5253```swift54let detection = FileTypeDetector.detect(fileURL)55if detection.mismatch { print("Wrong extension: \(detection.evidence)") }56let inspected = await manager.inspector.inspect(fileURL)57_ = inspected.inspection.pixelWidth          // nil when the engine could not read it58```5960- `DetectionResult.format` is `nil` when nothing identifies the file. `confidence` is `.certain` (signature or61  container structure read), `.likely` (text heuristics, extension-assisted rules) or `.extensionOnly`.62- `mismatch` is true when the name claims a format the content contradicts — or a format whose mandatory signature63  is absent. Legitimate overlaps are not mismatches (`FormatEquivalence`): an APNG named `.png`, ALAC in `.m4a`,64  `.tar.gz`, an SVG named `.xml`, loose text formats named after each other, etc.65- The detector reads at most 64 KiB from the start, plus bounded container reads (ZIP central directory ≤ 4 MiB and66  20 000 entries, OLE2 directory ≤ 32 sectors, `moov` ≤ 1 MiB). It never throws and never trusts a declared length.67- `FileInspector.inspect` returns the engine's `FileInspection` with a `File` group of generic fields prepended68  (size, dates, permissions, UTI, quarantine / where-from attribute presence), plus typed `FileFacts`.6970## Smart Drop7172```swift73let analysis = await manager.smartDrop.analyze(droppedURLs)74for group in analysis.groups {75    print(group.format?.rawValue ?? "unknown", group.urls.count, group.recommended.map(\.destination))76}77if let first = analysis.items.first {78    let (plan, warnings) = try manager.smartDrop.preflight(first, to: .markdown)79    print(plan.isChained, plan.fidelity, warnings.map(\.message))80}81```8283`analyze` expands dropped folders (hidden files and symlinks skipped, iWork packages kept whole, bounded by84`Configuration.maxFiles`, `truncated` says when the bound was hit), inspects with bounded concurrency, and returns85per-file targets/recommendations/operations/warnings, groups by format, the destinations common to the whole batch86and, for each dropped folder, what can be done with the folder itself (paths whose source is `.folder`).87`preflight` gives what must be shown before converting: plan limitations, the "two steps" notice for chained plans88and the engine's own `preflight` warnings.8990## Capabilities and plans9192```swift93let capabilities = manager.resolver.capabilities(for: .txt)94_ = capabilities.recommended.map(\.destination)95let plan = try manager.planner.plan(from: .txt, to: .html)96_ = (plan.isChained, plan.intermediates, plan.limitations)97```9899- Recommendation score (documented on `ConversionCapabilityResolver.score`): +100 same category, +60 − 5 × position100  in `commonTargets` for the source category (floor +10), +10 if `supported`, +4/+3/+2/+1 by fidelity.101- Planner: a direct path when one exists (engine chosen by status, then fidelity; `engine:` forces one). Otherwise,102  for plain conversions only, a chain of exactly two paths. Chains through a lossy intermediate are discarded when a103  lossless one exists; the rest are ranked by `stepCost` (0 lossless, 1 structural, 4 lossy, 8 partial, +2104  experimental). Chains never include a named operation, a folder intermediate or a multi-source path.105- A chained `ConversionPlan` has `isChained == true`, the union of the steps' limitations, the worst fidelity and106  status, and the intersection of `preserves`.107108## Observing the queue109110```swift111let events = await queue.events()112let observer = Task {113    for await event in events {114        switch event {115        case .jobAdded(let job), .jobChanged(let job): print(job.id, job.state)116        case .jobProgress(let id, let progress, let eta): print(id, progress.fraction ?? -1, eta ?? -1)117        case .jobLog(_, let line): print(line.date, line.level, line.message)118        case .jobWarning(_, let warning): print(warning.severity, warning.message)119        case .jobRemoved, .queueChanged, .idle: break120        }121    }122}123```124125Each call to `events()` returns an independent stream; cancel the consuming task to unsubscribe. A consumer that126falls more than 4096 events behind loses the oldest ones — call `snapshot()` to resynchronise.127`progress.fraction` is the engine's measured fraction mapped over the plan's steps, or `nil`.128`estimatedSecondsRemaining` is `elapsed × (1 − f) / f` and stays `nil` until a fraction ≥ 1 % has been measured.129130## Enqueueing131132```swift133let destination = OutputDestination(directory: outputFolder, nameTemplate: "{name}-{date}", conflictPolicy: .askUser)134let jobIDs = await manager.convert(droppedURLs, to: .markdown, destination: destination)135let single = await queue.enqueue(ConversionJobRequest(source: fileURL, to: .yaml, destination: destination, priority: .high))136```137138- One request = one job. `manager.convert` makes one job per file (a batch: shared `batchID`, `counter` 1…n);139  `manager.combine` makes a single job from several sources for paths with `acceptsMultipleSources`.140- `sourceFormat: nil` (the default) means "detect from content when the job starts".141- `OutputDestination.directory == nil` writes next to the source. Name tokens: `{name}`, `{ext}` (source142  extension), `{date}`, `{counter}` / `{counter:3}`, `{width}`, `{height}` (dropped when unknown; dimensions the143  engine reports in `ConversionOutput.facts["width"/"height"]` win over the source's).144- In a chained plan, `options` go to the last step; the first step runs with its defaults.145146What a job does: `preparing` (detect, plan, inspect, early conflict check) → `running` (each step in its own147`step-N` folder of a private work directory) → `validating` (generic checks from `OutputValidator`, then the148engine's `validate`; any failed check fails the job and nothing is delivered) → atomic move by `SafeWriter` →149`finished`. Every ended job is written to the history store.150151### Conflicts and the source file152153| Policy | Destination exists |154|---|---|155| `.rename` | writes `name 2.ext`, `name 3.ext`… |156| `.skip` | job ends `skipped` (checked before converting) |157| `.replace` | atomic swap (`replaceItemAt`) |158| `.askUser` | job goes to `awaitingDecision` with `job.conflict`; call `resolveConflict` |159160The output never overwrites one of the job's sources unless the policy is `.replace` **and**161`allowReplacingSource` is true. With `.rename` it is numbered; with `.askUser` the conflict has `isSource == true`.162163### Control164165```swift166await queue.pause()                       // no new job starts; running jobs finish167await queue.resume()168_ = await queue.pauseJob(single)          // only before the job starts169_ = await queue.resumeJob(single)170_ = await queue.setPriority(.low, for: single)171_ = await queue.cancel(single)172_ = await queue.retry(single)173```174175Pausing is honest: a **running** job cannot be paused (engines have no cooperative suspension point), so176`pauseJob` returns `false` for it. `cancel` cancels the job's task; engines stop at their next177`context.checkCancellation()` and `ProcessRunner` terminates helper processes.178Also available: `move(_:to:)`, `remove(_:)`, `clearSettled()`, `cancelAll()`, `updateConfiguration(_:)`.179180### Waiting and resolving181182```swift183for id in jobIDs + [single] {184    guard let job = await queue.waitForJob(id) else { continue }185    if job.state == .awaitingDecision, let conflict = job.conflict {186        print("Exists:", conflict.existing.path)187        _ = await queue.resolveConflict(id, policy: .rename)   // or .replace / .skip188    }189}190await queue.waitUntilIdle()191let snapshot = await queue.snapshot()192print(snapshot.status.active, snapshot.jobs.map(\.state))193```194195`waitForJob` returns when the job is *settled*: terminal, or waiting for a decision. When196`conflict.conversionDone` is true the converted file is held in its work directory and `resolveConflict` only moves197it; otherwise the conflict was found before converting and resolving starts the conversion.198199### Resource governance200201`QueueConfiguration.limits(for:)` is the whole rule: thermal `critical` → no new start; `serious` → half the202concurrency and one heavy job; Low Power Mode (`reduceInLowPowerMode`, default on) or battery (`reduceOnBattery`,203default off) → half. A job is *heavy* when its source or destination is a video format, its plan uses an engine204listed in `heavyEngineIDs`, or its sources reach `heavyFileSize` (1 GiB by default). `snapshot().status` exposes the205limits in force and the reason. Thermal and Low Power Mode changes reschedule automatically; the battery state is206re-read at every scheduling decision (call `resourceStateChanged()` to force one).207208## History209210```swift211var query = HistoryQuery()212query.search = "note"213query.statuses = ["finished"]214query.limit = 50215let entries = try await history.entries(query)216try await history.delete(ids: entries.map(\.id))217```218219SQLite in WAL mode. One row per ended job (paths, formats, operation, engine(s), options, sizes, duration, status,220warnings, validation checks, error, timestamps) and one row per workflow run (`workflowRuns`). File contents are221never stored. `count(_:)`, `clearAll()`, `deleteWorkflowRuns(ids:)` complete the API.222223## Presets224225```swift226let presets = PresetStore(fileURL: supportFolder.appendingPathComponent("presets.json"))227await presets.registerBuiltIns([ConversionPreset(name: "Markdown", source: .format(.txt), destinationFormat: .markdown, isBuiltIn: true)])228let preset = try await presets.add(ConversionPreset(name: "My YAML", source: .category(.document), destinationFormat: .yaml))229_ = await manager.apply(preset, to: droppedURLs, destination: OutputDestination(directory: outputFolder))230```231232The core ships **no** built-in preset: the app registers them at start-up with `registerBuiltIns` (they are233read-only and not written to the user's file). User presets: `add`, `update`, `delete`, `duplicate`,234`exportData(ids:)`, `importData(_:)`. A corrupt file is reported through `loadError` and left untouched until the235next save.236237## Workflows (Conversion Flow)238239```swift240let workflow = Workflow(name: "Small notes to Markdown", nodes: [241    .filter(.all([.categoryIs(.document), .sizeBelow(1_000_000)])),242    .branch(condition: .nameMatches("draft-*"), then: [.rename(template: "{name}-draft")], otherwise: []),243    .convert(destination: .markdown, operation: nil, options: ConversionOptions()),244    .export(folder: outputFolder, conflictPolicy: .rename),245])246let validation = WorkflowValidator.validate(workflow, registry: registry, inputFormats: [.txt])247for issue in validation.issues { print(issue.severity, issue.nodePath, issue.message) }248let library = WorkflowLibrary(fileURL: supportFolder.appendingPathComponent("workflows.json"))249try await library.save(workflow)250let run = try await manager.workflowRunner.run(workflow, inputs: droppedURLs) { item in251    print(item.source.lastPathComponent, item.status, item.outputs)252}253print(run.count(.succeeded), run.count(.failed), run.count(.filtered))254```255256- Nodes run in order on the file the workflow currently holds: `convert` replaces it by the conversion result (a257  queue job whose output stays in the run's work directory), `rename` sets the template for later exports, `filter`258  drops the item, `branch` runs one side then continues, `export` **copies** the current file to a folder. A path259  that ends without an export writes to `WorkflowRunOptions.defaultExportFolder` (default: next to the source), and260  writes nothing when the file is unchanged and would land in its own folder.261- A convert node producing several files makes the following nodes apply to each; conditions look at the first.262- `WorkflowIssue.nodePath` is 1-based and nested: `2`, `3.then.1`, `3.otherwise.2`.263- Validation with explicit `inputFormats` requires a real path for every format reaching each convert node; without264  them (as `run` does) a convert node is an error only when no format can reach its destination, and unsupported265  inputs fail individually at run time.266- `errorPolicy`: `.stopItem` ends the item at the failing node, `.continueItem` skips the node. Other items are267  never affected. Cancelling the task that called `run` cancels the outstanding jobs.268- `WorkflowLibrary` never stores a secret: option values whose key contains `password` are removed from convert269  nodes (also inside branches) on `save`, `importData`, `exportData` and when an older file is loaded270  (`Workflow.redactingSecrets()`). Run the in-memory workflow when a password is needed.271- A file already in the destination format is converted again only when the step changes a setting of a real272  same-format path (`SameFormatRule`).273- The validator assumes every node succeeds: under `.continueItem`, a node placed after a failing convert is still274  validated against that convert's destination format.275276## Watched folders277278```swift279let watcher = FolderWatcher(runner: manager.workflowRunner, presets: presets, workflows: library,280                            storeURL: supportFolder.appendingPathComponent("watch-rules.json"))281let rule = WatchRule(name: "Inbox to Markdown", folder: fileURL.deletingLastPathComponent(), match: .formats([.txt]),282                     action: .workflow(workflow.id), destinationFolder: outputFolder)283do {284    let added = try await watcher.addRule(rule)        // throws WatchError.loopRisk for a rule that feeds itself285    await watcher.start()286    try await watcher.setRulePaused(added.id, true)287    await watcher.stop()288} catch let error as WatchError {289    print(error.localizedDescription)290}291```292293- Only the folders of the rules are watched (FSEvents, plus a periodic re-listing in case an event is missed).294  Bookmarks are stored with each rule (security-scoped when the system grants one) and resolved at `start()`, so a295  renamed folder is followed.296- A file is processed once its size and modification date have been unchanged for297  `WatcherConfiguration.settleInterval` (2 s by default) and its name is not temporary (`isTemporaryName`).298  Only regular, non-empty files; folders dropped in a watched folder are ignored.299- Files already present when a rule starts are ignored unless `processExistingFiles` is set. The same holds for300  files that arrived while the app was not running or the rule was paused.301- Loop prevention: `addRule` refuses (`WatchError.loopRisk`) a rule whose destination is in its watched folder302  while its outputs can match it; and a file written by the watcher is never processed by any rule — two rules303  cannot be chained through a shared folder (use one workflow instead).304- Duplicates: unchanged path + size + date is skipped silently; identical content arriving again (SHA-256) is305  recorded as `.duplicate`. `skipDuplicateContent: false` disables the content check.306- `activity` (bounded, persisted) and `events()` feed the UI; `pauseAll()` / `resumeAll()` are global.307308## Compatibility matrix309310```swift311let markdown = registry.matrixMarkdown()312let json = try registry.matrixJSON()313print(registry.counts.verifiedPairs, markdown.count, json.count)314```315316`matrix()` returns one row per declared path (source, destination, operation, engine, status, fidelity, preserves,317limitations); paths of unavailable engines come last with status `unavailable` and the reason. `counts` gives the318real totals: formats with at least one path, distinct pairs, pairs with a `supported` path, operations.319320## Notes for engine authors321322- `context.outputURL(for:)` appends the destination's preferred extension. For `.folder` destinations323  `FileFormat.preferredExtension` falls back to the identifier, so the default call yields `name.folder`:324  pass `extension: ""` to get a directory named `name`.325- Name outputs from `request.outputBaseName` (several outputs: `outputBaseName + "-1"`…): the core derives final326  names from it.327- The generic validation runs the detector on every output file: content must be the destination format or an328  equivalent listed in `FormatEquivalence`. A format the detector has no signature for passes on its extension329  (the check says so).330- Intermediate steps of a chained plan must produce exactly one file.331332## Additions of the integration pass (2026-11-21)333334- **Container / parent formats** — `FormatEquivalence.children`, `isParent(_:of:)`, `parents(of:)`: zip ⊇ docx, xlsx,335  pptx, odt, ods, odp, epub, 3mf, usdz, kmz, pages, keynote; xml ⊇ svg, kml, gpx, dae, plist, html; json ⊇ geojson,336  gltf; sqlite ⊇ gpkg; gz ⊇ tgz… `accepts(detected:as:)` accepts a child where its parent was announced (never the337  reverse). `ttf ↔ otf` are equivalent (same sfnt container). A file **named `.zip`** whose entries look like one of338  the ZIP children is detected as `zip` (`certain`, no mismatch); other parents keep the precise child.339- **`kmz`** is a catalog format (ZIP with `doc.kml` or a top-level `.kml`). Zipped shapefiles are not detected.340- **Detector order** — the lenient "`%PDF-` within the first kilobyte" rule is the last binary rule, after every341  fixed-offset signature (TAR, ZIP, gzip, zstd… that store a small PDF first are no longer PDFs).342- **`ConversionPath`** gained two additive flags (default `false`, older stored values decode):343  `producesMultipleOutputs` and `appliesToAnyFile`.344- **Chains** — `ConversionPlanner.canChain`: plain single-source steps, the first cannot yield several files345  (`mayProduceSeveralFiles`: the flag, a folder destination, or a multi-page / multi-sheet source to a single-page346  image or drawing), and the two steps belong to **different engines**. `chainedDestinations` lists only what the347  planner accepts.348- **Generic operations** — `ConversionCapabilityResolver.genericOperations()`: `.folder` paths that accept several349  sources are offered for any selection of files and folders; `.folder` paths marked `appliesToAnyFile` for any single350  regular file. `SmartDropAnalysis.selectionOperations` / `.fileOperations`; `ConversionJobManager.enqueueGeneric`351  enqueues them with `sourceFormat: .folder`.352- **Several output files** — when a job's engine returns more than one item for a destination that is not a folder,353  they are delivered in ONE folder named like the single output would have been (`facts["outputFiles"]`), under the354  job's conflict policy. `ConversionJobRequest.groupMultipleOutputs = false` keeps loose files (workflows use it for355  their internal jobs; a workflow export of several files from one input goes to a folder named after the input).356- **Same-format conversions in workflows and watch rules** — `SameFormatRule.decide`: a convert step whose357  destination is the file's own format runs only when a same-format path exists **and** the step sets an option of358  that path to a non-default value; otherwise the file passes through (`settingsIgnored` adds the warning359  `same-format-settings-ignored` when settings were given and no such path exists). Named operations always run.360- **Folder outputs** — valid with at least one entry of any kind; a symbolic link is inside when its target path,361  folded lexically, stays inside (dangling targets included) and the existing part of it, links followed, stays inside.362- **Inspector choice** — `FormatRegistry.inspector(for:)` prefers the engine with a plain same-category conversion363  from the format, then one with any same-category path, then registration order.364- **`BundledTools`** — the repository fallback is off for the shipped application (main bundle is an `.app` with the365  identifier `ai.spboucher.orix.convert` or a `Contents/Helpers` folder, or the executable sits in366  `….app/Contents/Helpers`, whose directory is then searched). `swift test` (run from inside Xcode.app) keeps it.367- **Watch rules** never keep or store secret option values (`WatchRule.redactingSecrets()`, applied on add, update368  and load).369
Files main
Menu

Theme