SPB Git

Repository Public License

core-api.md 23.2 KB

# ORIX Convert — core API (ConvertCore)

Everything the app needs between "the user dropped files" and "the converted files are in place" lives in Packages/ORIXConvertCore (module ConvertCore). The core contains no conversion code: engines (Packages/Engine*) implement ConversionEngine; the core detects, plans, schedules, validates, writes and records.

All public types are Sendable. Stateful services are actors; everything else is a value type. Every sample below is compiled and executed by Tests/ConvertCoreTests/DocSamplesTests.swift.

# Map of the module

Area Types File
Contract (shared, do not change) FormatID, FileFormat, FormatCatalog, ConversionEngine, ConversionPath, ConversionRequest, ConversionContext, ConversionOutput, FileInspection, ValidationCheck, ConversionError, BundledTools, ProcessRunner… Formats.swift, Engine.swift, Tools.swift
Detection FileTypeDetector, DetectionResult, TextEncoding, FormatEquivalence Detection/
Registry FormatRegistry, RegistryCounts, MatrixRow, UnavailableEngine FormatRegistry.swift
Capabilities and planning ConversionCapabilityResolver, ConversionCapabilities, ConversionTarget, ConversionPlanner, ConversionPlan Planner.swift
Safe output WorkDirectoryManager, OutputValidator, SafeWriter, ConflictPolicy, CommittedOutput, OutputNameTemplate, OutputNameContext, AppPaths SafeOutput.swift
Queue ConversionJobManager, ConversionQueue, ConversionJobRequest, ConversionJob, JobState, JobPriority, OutputDestination, JobProgress, JobLogLine, DestinationConflict, QueueEvent, QueueSnapshot, QueueStatus, QueueConfiguration, ConcurrencyLimits, ResourceState, ThermalLevel, ResourceMonitoring, SystemResourceMonitor, ETAEstimator Jobs/
History ConversionHistoryStore, HistoryEntry, HistoryQuery, WorkflowRunRecord, HistoryError History.swift
Presets PresetStore, ConversionPreset, PresetError Presets.swift
Workflows Workflow, WorkflowNode, WorkflowCondition, Comparison, Glob, WorkflowErrorPolicy, WorkflowValidator, WorkflowValidation, WorkflowIssue, WorkflowRunner, WorkflowRunOptions, WorkflowRunResult, WorkflowItemResult, WorkflowLibrary, WorkflowError Workflows.swift
Watched folders FolderWatcher, WatchRule, WatchActivity, WatchEvent, WatcherConfiguration, WatchError FolderWatcher.swift
Inspection FileInspector, InspectedFile, FileFacts, SmartDropAnalyzer, SmartDropAnalysis, SmartDropItem, SmartDropGroup, SmartDropFolder Inspection.swift

# Setting up

swift
let history = try ConversionHistoryStore(url: supportFolder.appendingPathComponent("history.sqlite"))
let manager = ConversionJobManager(engines: engines, history: history,
                                   workDirectories: WorkDirectoryManager(root: temp.file("work")))
let registry = manager.registry
let queue = manager.queue

In the app, engines is EngineImage.makeEngines() + EngineMedia.makeEngines() + …, and the defaults place everything where it belongs: ConversionHistoryStore() opens ~/Library/Application Support/ORIX Convert/history.sqlite, WorkDirectoryManager() uses $TMPDIR/ai.spboucher.orix.convert/work (mode 0700; directories left by dead processes are swept when the queue is created). AppPaths lists the locations; ORIX_CONVERT_SUPPORT_DIR overrides the support folder for tools and tests.

The registry is a snapshot: engine availability (availability()) is read once, when it is built. Paths of an unavailable engine are excluded from registry.paths and listed in registry.unavailableEngines with the reason.

Put the heavy engines' identifiers in QueueConfiguration.heavyEngineIDs (the core cannot guess them):

swift
QueueConfiguration(maxConcurrentJobs: 6, maxHeavyJobs: 1, heavyEngineIDs: ["documents-office", "media"], reduceOnBattery: true)

# Detecting and inspecting

swift
let detection = FileTypeDetector.detect(fileURL)
if detection.mismatch { print("Wrong extension: \(detection.evidence)") }
let inspected = await manager.inspector.inspect(fileURL)
_ = inspected.inspection.pixelWidth          // nil when the engine could not read it
  • DetectionResult.format is nil when nothing identifies the file. confidence is .certain (signature or container structure read), .likely (text heuristics, extension-assisted rules) or .extensionOnly.
  • mismatch is true when the name claims a format the content contradicts — or a format whose mandatory signature is absent. Legitimate overlaps are not mismatches (FormatEquivalence): an APNG named .png, ALAC in .m4a, .tar.gz, an SVG named .xml, loose text formats named after each other, etc.
  • The detector reads at most 64 KiB from the start, plus bounded container reads (ZIP central directory ≤ 4 MiB and 20 000 entries, OLE2 directory ≤ 32 sectors, moov ≤ 1 MiB). It never throws and never trusts a declared length.
  • FileInspector.inspect returns the engine's FileInspection with a File group of generic fields prepended (size, dates, permissions, UTI, quarantine / where-from attribute presence), plus typed FileFacts.

# Smart Drop

swift
let analysis = await manager.smartDrop.analyze(droppedURLs)
for group in analysis.groups {
    print(group.format?.rawValue ?? "unknown", group.urls.count, group.recommended.map(\.destination))
}
if let first = analysis.items.first {
    let (plan, warnings) = try manager.smartDrop.preflight(first, to: .markdown)
    print(plan.isChained, plan.fidelity, warnings.map(\.message))
}

analyze expands dropped folders (hidden files and symlinks skipped, iWork packages kept whole, bounded by Configuration.maxFiles, truncated says when the bound was hit), inspects with bounded concurrency, and returns per-file targets/recommendations/operations/warnings, groups by format, the destinations common to the whole batch and, for each dropped folder, what can be done with the folder itself (paths whose source is .folder). preflight gives what must be shown before converting: plan limitations, the "two steps" notice for chained plans and the engine's own preflight warnings.

# Capabilities and plans

swift
let capabilities = manager.resolver.capabilities(for: .txt)
_ = capabilities.recommended.map(\.destination)
let plan = try manager.planner.plan(from: .txt, to: .html)
_ = (plan.isChained, plan.intermediates, plan.limitations)
  • Recommendation score (documented on ConversionCapabilityResolver.score): +100 same category, +60 − 5 × position in commonTargets for the source category (floor +10), +10 if supported, +4/+3/+2/+1 by fidelity.
  • Planner: a direct path when one exists (engine chosen by status, then fidelity; engine: forces one). Otherwise, for plain conversions only, a chain of exactly two paths. Chains through a lossy intermediate are discarded when a lossless one exists; the rest are ranked by stepCost (0 lossless, 1 structural, 4 lossy, 8 partial, +2 experimental). Chains never include a named operation, a folder intermediate or a multi-source path.
  • A chained ConversionPlan has isChained == true, the union of the steps' limitations, the worst fidelity and status, and the intersection of preserves.

# Observing the queue

swift
let events = await queue.events()
let observer = Task {
    for await event in events {
        switch event {
        case .jobAdded(let job), .jobChanged(let job): print(job.id, job.state)
        case .jobProgress(let id, let progress, let eta): print(id, progress.fraction ?? -1, eta ?? -1)
        case .jobLog(_, let line): print(line.date, line.level, line.message)
        case .jobWarning(_, let warning): print(warning.severity, warning.message)
        case .jobRemoved, .queueChanged, .idle: break
        }
    }
}

Each call to events() returns an independent stream; cancel the consuming task to unsubscribe. A consumer that falls more than 4096 events behind loses the oldest ones — call snapshot() to resynchronise. progress.fraction is the engine's measured fraction mapped over the plan's steps, or nil. estimatedSecondsRemaining is elapsed × (1 − f) / f and stays nil until a fraction ≥ 1 % has been measured.

# Enqueueing

swift
let destination = OutputDestination(directory: outputFolder, nameTemplate: "{name}-{date}", conflictPolicy: .askUser)
let jobIDs = await manager.convert(droppedURLs, to: .markdown, destination: destination)
let single = await queue.enqueue(ConversionJobRequest(source: fileURL, to: .yaml, destination: destination, priority: .high))
  • One request = one job. manager.convert makes one job per file (a batch: shared batchID, counter 1…n); manager.combine makes a single job from several sources for paths with acceptsMultipleSources.
  • sourceFormat: nil (the default) means "detect from content when the job starts".
  • OutputDestination.directory == nil writes next to the source. Name tokens: {name}, {ext} (source extension), {date}, {counter} / {counter:3}, {width}, {height} (dropped when unknown; dimensions the engine reports in ConversionOutput.facts["width"/"height"] win over the source's).
  • In a chained plan, options go to the last step; the first step runs with its defaults.

What a job does: preparing (detect, plan, inspect, early conflict check) → running (each step in its own step-N folder of a private work directory) → validating (generic checks from OutputValidator, then the engine's validate; any failed check fails the job and nothing is delivered) → atomic move by SafeWriter → finished. Every ended job is written to the history store.

# Conflicts and the source file

Policy Destination exists
.rename writes name 2.ext, name 3.ext…
.skip job ends skipped (checked before converting)
.replace atomic swap (replaceItemAt)
.askUser job goes to awaitingDecision with job.conflict; call resolveConflict

The output never overwrites one of the job's sources unless the policy is .replace and allowReplacingSource is true. With .rename it is numbered; with .askUser the conflict has isSource == true.

# Control

swift
await queue.pause()                       // no new job starts; running jobs finish
await queue.resume()
_ = await queue.pauseJob(single)          // only before the job starts
_ = await queue.resumeJob(single)
_ = await queue.setPriority(.low, for: single)
_ = await queue.cancel(single)
_ = await queue.retry(single)

Pausing is honest: a running job cannot be paused (engines have no cooperative suspension point), so pauseJob returns false for it. cancel cancels the job's task; engines stop at their next context.checkCancellation() and ProcessRunner terminates helper processes. Also available: move(_:to:), remove(_:), clearSettled(), cancelAll(), updateConfiguration(_:).

# Waiting and resolving

swift
for id in jobIDs + [single] {
    guard let job = await queue.waitForJob(id) else { continue }
    if job.state == .awaitingDecision, let conflict = job.conflict {
        print("Exists:", conflict.existing.path)
        _ = await queue.resolveConflict(id, policy: .rename)   // or .replace / .skip
    }
}
await queue.waitUntilIdle()
let snapshot = await queue.snapshot()
print(snapshot.status.active, snapshot.jobs.map(\.state))

waitForJob returns when the job is settled: terminal, or waiting for a decision. When conflict.conversionDone is true the converted file is held in its work directory and resolveConflict only moves it; otherwise the conflict was found before converting and resolving starts the conversion.

# Resource governance

QueueConfiguration.limits(for:) is the whole rule: thermal critical → no new start; serious → half the concurrency and one heavy job; Low Power Mode (reduceInLowPowerMode, default on) or battery (reduceOnBattery, default off) → half. A job is heavy when its source or destination is a video format, its plan uses an engine listed in heavyEngineIDs, or its sources reach heavyFileSize (1 GiB by default). snapshot().status exposes the limits in force and the reason. Thermal and Low Power Mode changes reschedule automatically; the battery state is re-read at every scheduling decision (call resourceStateChanged() to force one).

# History

swift
var query = HistoryQuery()
query.search = "note"
query.statuses = ["finished"]
query.limit = 50
let entries = try await history.entries(query)
try await history.delete(ids: entries.map(\.id))

SQLite in WAL mode. One row per ended job (paths, formats, operation, engine(s), options, sizes, duration, status, warnings, validation checks, error, timestamps) and one row per workflow run (workflowRuns). File contents are never stored. count(_:), clearAll(), deleteWorkflowRuns(ids:) complete the API.

# Presets

swift
let presets = PresetStore(fileURL: supportFolder.appendingPathComponent("presets.json"))
await presets.registerBuiltIns([ConversionPreset(name: "Markdown", source: .format(.txt), destinationFormat: .markdown, isBuiltIn: true)])
let preset = try await presets.add(ConversionPreset(name: "My YAML", source: .category(.document), destinationFormat: .yaml))
_ = await manager.apply(preset, to: droppedURLs, destination: OutputDestination(directory: outputFolder))

The core ships no built-in preset: the app registers them at start-up with registerBuiltIns (they are read-only and not written to the user's file). User presets: add, update, delete, duplicate, exportData(ids:), importData(_:). A corrupt file is reported through loadError and left untouched until the next save.

# Workflows (Conversion Flow)

swift
let workflow = Workflow(name: "Small notes to Markdown", nodes: [
    .filter(.all([.categoryIs(.document), .sizeBelow(1_000_000)])),
    .branch(condition: .nameMatches("draft-*"), then: [.rename(template: "{name}-draft")], otherwise: []),
    .convert(destination: .markdown, operation: nil, options: ConversionOptions()),
    .export(folder: outputFolder, conflictPolicy: .rename),
])
let validation = WorkflowValidator.validate(workflow, registry: registry, inputFormats: [.txt])
for issue in validation.issues { print(issue.severity, issue.nodePath, issue.message) }
let library = WorkflowLibrary(fileURL: supportFolder.appendingPathComponent("workflows.json"))
try await library.save(workflow)
let run = try await manager.workflowRunner.run(workflow, inputs: droppedURLs) { item in
    print(item.source.lastPathComponent, item.status, item.outputs)
}
print(run.count(.succeeded), run.count(.failed), run.count(.filtered))
  • Nodes run in order on the file the workflow currently holds: convert replaces it by the conversion result (a queue job whose output stays in the run's work directory), rename sets the template for later exports, filter drops the item, branch runs one side then continues, export copies the current file to a folder. A path that ends without an export writes to WorkflowRunOptions.defaultExportFolder (default: next to the source), and writes nothing when the file is unchanged and would land in its own folder.
  • A convert node producing several files makes the following nodes apply to each; conditions look at the first.
  • WorkflowIssue.nodePath is 1-based and nested: 2, 3.then.1, 3.otherwise.2.
  • Validation with explicit inputFormats requires a real path for every format reaching each convert node; without them (as run does) a convert node is an error only when no format can reach its destination, and unsupported inputs fail individually at run time.
  • errorPolicy: .stopItem ends the item at the failing node, .continueItem skips the node. Other items are never affected. Cancelling the task that called run cancels the outstanding jobs.
  • WorkflowLibrary never stores a secret: option values whose key contains password are removed from convert nodes (also inside branches) on save, importData, exportData and when an older file is loaded (Workflow.redactingSecrets()). Run the in-memory workflow when a password is needed.
  • A file already in the destination format is converted again only when the step changes a setting of a real same-format path (SameFormatRule).
  • The validator assumes every node succeeds: under .continueItem, a node placed after a failing convert is still validated against that convert's destination format.

# Watched folders

swift
let watcher = FolderWatcher(runner: manager.workflowRunner, presets: presets, workflows: library,
                            storeURL: supportFolder.appendingPathComponent("watch-rules.json"))
let rule = WatchRule(name: "Inbox to Markdown", folder: fileURL.deletingLastPathComponent(), match: .formats([.txt]),
                     action: .workflow(workflow.id), destinationFolder: outputFolder)
do {
    let added = try await watcher.addRule(rule)        // throws WatchError.loopRisk for a rule that feeds itself
    await watcher.start()
    try await watcher.setRulePaused(added.id, true)
    await watcher.stop()
} catch let error as WatchError {
    print(error.localizedDescription)
}
  • Only the folders of the rules are watched (FSEvents, plus a periodic re-listing in case an event is missed). Bookmarks are stored with each rule (security-scoped when the system grants one) and resolved at start(), so a renamed folder is followed.
  • A file is processed once its size and modification date have been unchanged for WatcherConfiguration.settleInterval (2 s by default) and its name is not temporary (isTemporaryName). Only regular, non-empty files; folders dropped in a watched folder are ignored.
  • Files already present when a rule starts are ignored unless processExistingFiles is set. The same holds for files that arrived while the app was not running or the rule was paused.
  • Loop prevention: addRule refuses (WatchError.loopRisk) a rule whose destination is in its watched folder while its outputs can match it; and a file written by the watcher is never processed by any rule — two rules cannot be chained through a shared folder (use one workflow instead).
  • Duplicates: unchanged path + size + date is skipped silently; identical content arriving again (SHA-256) is recorded as .duplicate. skipDuplicateContent: false disables the content check.
  • activity (bounded, persisted) and events() feed the UI; pauseAll() / resumeAll() are global.

# Compatibility matrix

swift
let markdown = registry.matrixMarkdown()
let json = try registry.matrixJSON()
print(registry.counts.verifiedPairs, markdown.count, json.count)

matrix() returns one row per declared path (source, destination, operation, engine, status, fidelity, preserves, limitations); paths of unavailable engines come last with status unavailable and the reason. counts gives the real totals: formats with at least one path, distinct pairs, pairs with a supported path, operations.

# Notes for engine authors

  • context.outputURL(for:) appends the destination's preferred extension. For .folder destinations FileFormat.preferredExtension falls back to the identifier, so the default call yields name.folder: pass extension: "" to get a directory named name.
  • Name outputs from request.outputBaseName (several outputs: outputBaseName + "-1"…): the core derives final names from it.
  • The generic validation runs the detector on every output file: content must be the destination format or an equivalent listed in FormatEquivalence. A format the detector has no signature for passes on its extension (the check says so).
  • Intermediate steps of a chained plan must produce exactly one file.

# Additions of the integration pass (2026-11-21)

  • Container / parent formats — FormatEquivalence.children, isParent(_:of:), parents(of:): zip ⊇ docx, xlsx, pptx, odt, ods, odp, epub, 3mf, usdz, kmz, pages, keynote; xml ⊇ svg, kml, gpx, dae, plist, html; json ⊇ geojson, gltf; sqlite ⊇ gpkg; gz ⊇ tgz… accepts(detected:as:) accepts a child where its parent was announced (never the reverse). ttf ↔ otf are equivalent (same sfnt container). A file named .zip whose entries look like one of the ZIP children is detected as zip (certain, no mismatch); other parents keep the precise child.
  • kmz is a catalog format (ZIP with doc.kml or a top-level .kml). Zipped shapefiles are not detected.
  • Detector order — the lenient "%PDF- within the first kilobyte" rule is the last binary rule, after every fixed-offset signature (TAR, ZIP, gzip, zstd… that store a small PDF first are no longer PDFs).
  • ConversionPath gained two additive flags (default false, older stored values decode): producesMultipleOutputs and appliesToAnyFile.
  • Chains — ConversionPlanner.canChain: plain single-source steps, the first cannot yield several files (mayProduceSeveralFiles: the flag, a folder destination, or a multi-page / multi-sheet source to a single-page image or drawing), and the two steps belong to different engines. chainedDestinations lists only what the planner accepts.
  • Generic operations — ConversionCapabilityResolver.genericOperations(): .folder paths that accept several sources are offered for any selection of files and folders; .folder paths marked appliesToAnyFile for any single regular file. SmartDropAnalysis.selectionOperations / .fileOperations; ConversionJobManager.enqueueGeneric enqueues them with sourceFormat: .folder.
  • Several output files — when a job's engine returns more than one item for a destination that is not a folder, they are delivered in ONE folder named like the single output would have been (facts["outputFiles"]), under the job's conflict policy. ConversionJobRequest.groupMultipleOutputs = false keeps loose files (workflows use it for their internal jobs; a workflow export of several files from one input goes to a folder named after the input).
  • Same-format conversions in workflows and watch rules — SameFormatRule.decide: a convert step whose destination is the file's own format runs only when a same-format path exists and the step sets an option of that path to a non-default value; otherwise the file passes through (settingsIgnored adds the warning same-format-settings-ignored when settings were given and no such path exists). Named operations always run.
  • Folder outputs — valid with at least one entry of any kind; a symbolic link is inside when its target path, folded lexically, stays inside (dangling targets included) and the existing part of it, links followed, stays inside.
  • Inspector choice — FormatRegistry.inspector(for:) prefers the engine with a plain same-category conversion from the format, then one with any same-category path, then registration order.
  • BundledTools — the repository fallback is off for the shipped application (main bundle is an .app with the identifier ai.spboucher.orix.convert or a Contents/Helpers folder, or the executable sits in ….app/Contents/Helpers, whose directory is then searched). swift test (run from inside Xcode.app) keeps it.
  • Watch rules never keep or store secret option values (WatchRule.redactingSecrets(), applied on add, update and load).
Files main
Menu

Theme