SPB Git

Repository Public License

implementation-status.md 9.9 KB 145 lines markdown

1<!--2ORIX OS Automation3File: docs/implementation-status.md4Author: Simon-Pierre Boucher5Contact: contact@spboucher.ai6-->7# Implementation status — 1.0.0 (1), 2026-11-2189Built and tested on one Mac: macOS 27.0, Apple Silicon, Xcode 27.2. "Verified" below means it was run for real on10that Mac on this date. Anything not listed as verified is not claimed.1112## Milestones1314| # | Milestone | State |15|---|---|---|16| M0 | Audit, contracts, credential broker, threat model, scripts | Done |17| M1 | App shell, design system, onboarding, settings, Keychain, store, console, permission center | Done; app launched and driven by UI tests |18| M2 | Three providers, agent loop, budgets, cancel, approvals | Done; live smoke tests passed for all three providers |19| M3 | Core macOS tools (77) | Done; integration-tested per family |20| M4 | Workflow engine, Workflow Studio, Code Studio | Done; live gestures in a real window not exercised |21| M5 | Scheduler, background agent, queue, export/import | Engine done and tested with a manual clock; **background agent registration and an unattended scheduled run were not exercised end to end** |22| M6 | Delegation, memory, skills, skill proposal pipeline | Done; the pipeline was run through services and the UI model, **not by a live model proposing a skill** |23| M7 | Watch & Learn, accessibility and performance passes | Recorder and conversion built and unit-tested; **never used on a real recording; no VoiceOver pass** |24| M8 | Regression, security review, docs, signed and notarized release | Done: app and DMG notarized, stapled, accepted by Gatekeeper |2526## Tests (2026-11-21)2728`Scripts/test/with-dev-keys.sh Scripts/test/test-all.sh` → all 18 packages passed, 789 tests29(`docs/release/test-report.json`). `xcodebuild test` → 2 UI tests passed against the real app.3031| Package | Tests | Kind |32|---|---|---|33| ORIXCore | 20 | unit |34| ORIXSecurity | 33 | unit + real Keychain (test-only service) + real audit files |35| ORIXPersistence | 24 | real SQLite, including corruption recovery and two connections |36| ORIXProviderAdapters | 68 | 57 fixture (mocked HTTP) + 11 live (3 providers) |37| ORIXToolRegistry | 115 | 36 unit, 63 integration, 16 security |38| ORIXAppleAutomation | 53 | 21 unit, 32 integration (osascript, AX, ScreenCaptureKit, Vision) |39| ORIXBrowserAutomation | 25 | real WKWebView against a loopback site |40| ORIXScheduler | 71 | cron/DST unit tests, real FSEvents, manual-clock engine |41| ORIXWorkflowEngine | 166 | codec, expressions, validator, engine (mock tools), graph, examples |42| ORIXAgentRuntime | 25 | scripted (mock) provider, real temp files |43| ORIXSkills | 9 | real confined processes |44| ORIXServices | 20 | the 15 examples end to end + headless refusals |45| ORIXAppUI, ORIXFeatureChat/Workflow/Code/Runs/Manage | 160 | view models on real services, render tests, performance measurements |46| App (XCUITest) | 2 | onboarding on first launch, all 11 sections, command palette |4748Mocked versus real is stated in each suite name (`Fixture`, `Mock provider`, `Integration`, `Live`).4950### Live provider status5152| Provider | Model | Result |53|---|---|---|54| Anthropic | `claude-haiku-5-5` | list models, streamed text, tool round trip, tool round trip with thinking: pass |55| OpenAI | `gpt-4o-mini` | list models, streamed text, tool round trip: pass |56| OpenRouter | `openai/gpt-4o-mini` | list models, streamed text, tool round trip, cost reporting: pass |5758Only these models were exercised live. Prompt caching, structured outputs, images, rate-limit handling and the retry59wrapper have mocked tests only. Cost of one full live pass: under US$0.02 including the agent examples.6061### End-to-end runs verified6263- `orixctl agent run` with Anthropic Haiku: read a CSV, wrote a result file, runtime-verified content (3 turns, $0.004), twice.64- 13 of 15 example workflows by real runs (files, PDFs, images, archive and restore, Python in the sandbox, git,65  HTTP on loopback, WebKit data entry, window capture, three with a live agent step).66- Headless run without accepted permissions ends `WAITING_APPROVAL` and changes nothing (3 examples).67- The app: launched (Debug and the notarized Release build), every section opened, real run history and workflows shown.6869### Skipped or not verified7071- **Calendar example**: Automation consent for Calendar was never given on this Mac; the run was not attempted to avoid triggering the prompt.72- **Notifications**: never seen delivered (tests and `orixctl` have no app bundle; in the app it was not exercised).73- **Background agent**: the launchd plist is bundled and the process starts and exits correctly when the setting is off; `SMAppService` registration and a schedule firing with the app closed were not exercised.74- **Accessibility actions**: `press` verified live; `setValue`, `focus` and successful `typeText` only through refusal paths and dry runs.75- **Watch & Learn**: no real recording session.76- **Intel slice** (built, never run), **macOS 15–26** (never run), **a second Mac or clean user account** (DMG never opened elsewhere).77- **VoiceOver**: labels and identifiers are in place; nobody listened to the app.78- **French**: about 1,800 strings translated; roughly 95 strings flagged by `Scripts/localize.py --missing` remain English (some are false positives); not proofread by a person.79- Cold-launch time and memory footprint were not measured. Measured: canvas frame time with 200 nodes (5–10 ms), editor keystroke latency on 5,000 lines (0.3 ms), console with 50,000 lines (11 ms per 500-line burst).8081## Security test outcomes8283| Threat scenario | Test | Outcome |84|---|---|---|85| 1 Web page asks to upload secrets | protected-path denial, network-grant and browser navigation-block tests | pass |86| 2 Fake system instructions in a document | untrusted framing, wrapper cannot be closed from inside | pass |87| 3 Skill exceeds its manifest | undeclared write and network denied by the sandbox; analyser blocker | pass |88| 4 Shell injection in tool output | literal argv, inline-interpreter refusal, marker file never created | pass |89| 5 Retrying an irreversible action | repeat of a succeeded tier ≥ 2 call blocked; non-idempotent steps not auto-retried | pass |90| 6 Invented success | status from runtime checks; claimed artifacts inspected | pass |91| 7 Key in console output | redaction of registered and key-shaped strings in results, events, receipts | pass |92| 8 Focus shift before input | re-resolve and frontmost check before each action; unfocused typing refused | pass (refusal paths) |93| 9 Stale permission after restart | expiry/revocation at every call; in-flight side effect → `RECOVERY_REQUIRED` | pass |94| 10 Hostile remote content | cross-host redirects not followed without a new decision (HTTP, browser, downloads) | pass |9596Release scan: no credential-shaped string in non-test sources or history, none of the real development keys in the97repository, its history or the bundle, no development credential adapter and no internal host name in the binary.9899## Known defects and gaps100101| Item | Severity |102|---|---|103| `secret("ref")` in workflow expressions produces a placeholder that no tool resolves; secrets work only through tools with explicit secret arguments (`http.request` secret headers, `browser.fill` secretRef) | Medium |104| Tool output and subprocess output are not persisted: a finished run shows summaries and events, not full output; workflow tool steps do not stream to the live console | Medium |105| Values of workflow `output:` steps are not stored in the receipt | Medium |106| Retry in Runs is hidden for parameterised workflows without a checkpoint (parameters are not on the run record) | Low |107| Skill drafts are kept in memory and lost on quit | Low |108| A workflow run that finishes within milliseconds may not expose step outputs in the Studio | Low |109| Seatbelt cannot filter network by host; the embedded browser does not filter subresources | Documented limit |110| Static analysis of scripts and skills is heuristic | Documented limit |111| Dry runs of two examples (backup, website monitor) end `FAILED` because later steps need files earlier steps would create | Low |112| The chat transcript showed an unexplained vertical gap in one off-screen render; not checked in the running app | Low |113114## Not built115116- Automatic updates (candidate and reasoning in `docs/release/runbook.md`).117- OpenAI Responses API; server-side tools of any provider.118- Vision-guided clicking (screen capture, OCR and image comparison exist; nothing clicks from pixels).119- Code Studio: debugger for scripts, multi-file projects, source-control view, formatters and validation for script languages.120- Cancelling a queued scheduled item from the UI; "run now" through the scheduler queue.121- Per-app Automation request button (macOS prompts on first use instead).122- Mac App Store build (not targeted; see the environment audit).123- Continuous-integration workflow files.124125## Human decisions and approvals outstanding1261271. Publishing: the repository has no remote and the DMG has not been uploaded anywhere.1282. Enable the background agent once in the installed app to exercise `SMAppService` and a closed-window scheduled run.1293. Grant Automation for Calendar if the calendar example should be verified.1304. A VoiceOver pass and a French proofreading pass.1315. Whether to add Sparkle (first third-party dependency) for updates.1326. The development keys used for testing are the SP Chat deployment's keys; testing spent a few cents.133134## Release135136| Item | Value |137|---|---|138| Artifact | `dist/ORIX-OS-Automation-1.0.0.dmg` (18 MB), `dist/ORIX-OS-Automation-1.0.0.zip` |139| Signature | Developer ID Application, team 3YM54G49SN, Hardened Runtime, secure timestamp |140| Architectures | arm64 + x86_64 |141| Entitlements | `com.apple.security.automation.apple-events` only |142| Notarization | app `54cc40eb-6138-45d6-ac41-0254116a7a31` Accepted; DMG `0b78f699-f6b9-437c-b8e6-a495059674e3` Accepted; both stapled |143| Gatekeeper | app and DMG: accepted, source = Notarized Developer ID |144| Evidence | `docs/release/release-1.0.0-1.md` (generated from the tools' output) |145
Files main
Menu

Theme