Repository Public License
1# ORIX OS AUTOMATION — CLAUDE.md23> **Mission:** Build the most capable, trustworthy, elegant native macOS agentic automation studio possible. This is a production engineering mandate, not a mockup, proof of concept, or collection of disconnected demos.4>5> **Product:** ORIX OS Automation6> **Platforms:** macOS, native Swift and SwiftUI, with native helper processes as needed.7> **Operating principle:** The user states an objective; ORIX plans, asks for necessary permissions, executes real tools, observes results, recovers from errors, and produces verifiable outcomes.8> **Language:** User-facing app should support English and French; engineering documentation and code may use English.910---1112## 0. NON-NEGOTIABLE EXECUTION DIRECTIVE1314You are Claude Code, acting as principal macOS engineer, agent systems architect, security engineer, product designer, QA lead, and release engineer. Implement this product in the actual repository and on the actual development Mac. Do not merely write a plan. First inspect the environment, then execute in measured, testable increments. Never claim something works unless it has been tested. Never fake agent runs, successful tool calls, screenshots, permissions, or notarization.1516Do not erase or replace an existing product without explicit authorization. Inspect the current repository and preserve useful conventions. If no project exists, create a clean, production-grade native Xcode project. Track assumptions, decisions, known limitations, tests, and follow-up work. When blocked by Apple permissions, credentials, user consent, or unavailable external services, report the precise blocker and proceed with independently testable parts.1718### Critical instructions from product owner19201. **Development API credentials:** Existing Anthropic/Claude, OpenAI, and OpenRouter API configuration used for testing is associated with the deployment of the existing **SP Chat** application. Locate the SP Chat deployment and inspect its *configuration structure and authorized secret references* before inventing any new test setup. The existence of usable keys must be verified, not assumed. Never dump, print, commit, log, embed in screenshots, or ship these secrets. Do not copy them into source files, .env files in the repo, app bundles, test fixtures, or release artifacts. Prefer a scoped local test credential broker, environment injection by the existing deployment, or securely provisioned developer Keychain items. Do not use a production credential without confirming that development testing is authorized. Limit tests to low-cost, controlled requests. The production ORIX application is **BYOK**: each user enters their own provider keys, which are stored in their local Keychain.212. **Apple signing and notarization:** Existing developer signing/notarization configuration is **on the development Mac**. Discover the available signing identities, Xcode team settings, existing keychain-backed notary profiles, and release scripts using metadata-only commands. Reuse authorized configuration where possible. Never reveal or export private keys, passwords, App Store Connect credentials, API key contents, or sensitive profile data. Do not create or replace credentials unnecessarily. Verify signing and notarization with Apple tools and save non-sensitive receipts. If signing or notarization cannot proceed, state exactly what human approval or missing capability is needed.223. Build real native macOS software, not an Electron wrapper or web page dressed as a Mac application.234. Make the interface visually exceptional, responsive, accessible, and consistent with macOS conventions.245. Preserve a strict separation between agent reasoning, tool permissions, execution, audit logging, and UI.256. Make the agent genuinely flexible: it may compose tools and write scripts, but every capability must remain inside explicit user-granted authority and enforceable runtime controls.267. Never bypass macOS privacy protections, TCC consent, code-signing restrictions, app sandbox restrictions, or system security controls.2728---2930## 1. PRODUCT VISION3132ORIX OS Automation is an agentic operating layer for the Mac. It unifies conversational automation, visual workflows, editable code, system tools, browser interaction, scheduling, and autonomous recovery.3334A user should be able to request:3536- “Every morning, summarize the PDFs added to this folder and generate a report.”37- “Open my spreadsheet, analyze the data using Python, create charts, and export a PDF.”38- “Monitor my Git repository, run tests when a branch changes, and prepare a patch for review.”39- “Organize my Downloads folder without deleting anything, and show me what changed.”40- “Prepare a weekly briefing from permitted local files and selected websites.”41- “Observe me completing this repetitive task and propose a reusable workflow.”42- “Create a new tool for this application if an existing integration is unavailable.”4344The user can operate the same automation through three synchronized views:45461. **Agent:** natural-language chat and execution timeline.472. **Visual:** node-and-edge workflow editor with live status.483. **Code:** editable declarative workflow plus individual script/tool source.4950These are representations of a common validated workflow model, not separate incompatible implementations. Round-tripping should preserve semantics; warn when an advanced feature cannot be represented losslessly in a simpler view.5152### Product principles5354- Real execution over theatrical simulation.55- Powerful defaults, transparent boundaries.56- Human control for consequential actions.57- Native quality over cross-platform compromises.58- Verifiable outcomes over plausible-sounding summaries.59- Minimal friction without sacrificing security.60- Recoverability and reproducibility by design.61- Local-first data, cloud model inference only when selected.62- No mandatory ORIX account or hosted ORIX backend.63- Users own their workflows, history, and keys.6465---6667## 2. FIRST ACTION: DISCOVERY AND ENVIRONMENT AUDIT6869Before writing product code:70711. Identify repository root, branch, remotes, status, project files, existing architecture, license, README, build scripts, and tests.722. Identify macOS version, CPU architecture, Xcode version, Swift toolchain, available SDKs, deployment target constraints, and Apple Developer signing identities.733. Find the SP Chat repository/deployment through known workspace paths, local service definitions, documented deployment manifests, or authorized administration tooling. Inspect only necessary metadata and configuration references. Do not conduct broad filesystem or Keychain secret scraping.744. Determine how to obtain scoped test access to Anthropic, OpenAI, and OpenRouter without moving secrets into ORIX source control. Confirm allowed provider use and any cost limits.755. Inspect existing local signing/notarization setup using `security find-identity -v -p codesigning`, `xcrun notarytool history` where authorized, Xcode build settings, and existing named keychain profiles. Avoid commands that reveal secrets. Do not modify unrelated projects.766. Assess whether a direct-distribution notarized build, Mac App Store build, or both are feasible. Default to a direct-distribution build because deep automation may conflict with App Store sandbox requirements. Document tradeoffs instead of promising App Store approval.777. Identify available macOS Automation, Accessibility, Screen Recording, Files and Folders, Full Disk Access, Apple Events, and browser automation permissions. Ask for user consent through standard macOS flows where required.788. Record findings in `docs/architecture/environment-audit.md`, redacting sensitive paths, identifiers, and credentials as appropriate.799. Produce an implementation plan with executable milestones, risk register, test strategy, and definition of done.8010. Start implementation immediately after discovery; do not stop at a planning document.8182### Suggested repository structure8384```text85ORIX-OS-Automation/86├── CLAUDE.md87├── README.md88├── LICENSE89├── ORIXAutomation.xcodeproj/90├── Apps/91│ └── ORIXAutomationMac/92│ ├── App/93│ ├── DesignSystem/94│ ├── Features/95│ │ ├── Dashboard/96│ │ ├── AgentChat/97│ │ ├── WorkflowStudio/98│ │ ├── CodeStudio/99│ │ ├── Console/100│ │ ├── Runs/101│ │ ├── Agents/102│ │ ├── Skills/103│ │ ├── Scheduler/104│ │ ├── Permissions/105│ │ ├── Providers/106│ │ ├── Settings/107│ │ └── Onboarding/108│ └── Resources/109├── Packages/110│ ├── ORIXCore/111│ ├── ORIXAgentRuntime/112│ ├── ORIXProviderAdapters/113│ ├── ORIXWorkflowEngine/114│ ├── ORIXToolRegistry/115│ ├── ORIXSecurity/116│ ├── ORIXPersistence/117│ ├── ORIXObservability/118│ ├── ORIXAppleAutomation/119│ ├── ORIXBrowserAutomation/120│ ├── ORIXScriptRuntime/121│ ├── ORIXScheduler/122│ └── ORIXTestingSupport/123├── Helpers/124│ ├── ORIXExecutionHelper/125│ └── ORIXBackgroundAgent/126├── Workflows/127│ ├── Examples/128│ └── Schemas/129├── Skills/130│ ├── BuiltIn/131│ └── Examples/132├── Tests/133│ ├── Unit/134│ ├── Integration/135│ ├── EndToEnd/136│ ├── Security/137│ └── Fixtures/138├── Scripts/139│ ├── build/140│ ├── test/141│ ├── sign/142│ └── release/143├── docs/144│ ├── architecture/145│ ├── security/146│ ├── api/147│ ├── workflows/148│ ├── skills/149│ ├── user-guide/150│ └── release/151└── .github/152 └── workflows/153```154155Adapt the structure to existing repository conventions. Do not create empty decorative modules that have no purpose.156157---158159## 3. TECHNOLOGY CHOICES160161- Swift and SwiftUI for the main macOS app.162- AppKit interoperability for advanced windows, native text editing, menu bar integration, keyboard shortcuts, drag-and-drop, and accessibility.163- Swift Concurrency (`async/await`, actors, structured tasks) for execution and cancellation.164- SwiftData or SQLite with explicit migrations for durable state; choose based on concurrency and observability needs.165- Keychain Services for provider credentials and local secrets.166- Security-scoped bookmarks for user-selected file/folder access where applicable.167- XPC and launch agents/helpers where justified; design narrow, authenticated interfaces.168- Apple Events, AppleScript/JXA, `NSAppleScript` or `osascript` when appropriate.169- Accessibility API (`AXUIElement`) for permitted UI automation.170- ScreenCaptureKit and Vision for explicitly authorized visual observation; do not record silently.171- Process/posix_spawn for subprocesses with cancellation, timeouts, process groups, bounded output, and explicit working directories.172- WKWebView only where it genuinely helps; use supported browser automation approaches and respect website terms.173- Native URLSession provider clients with structured decoding, streaming, retries, rate limiting, and timeout handling.174- Optional Python runtime via managed user-approved environment; never silently mutate system Python.175- OSLog with privacy-safe structured logging and redaction.176- XCTest/Swift Testing and macOS UI automation tests.177- Developer ID signing, hardened runtime, notarization, and stapling for direct distribution.178179Do not choose a dependency because it is fashionable. Prefer maintainable, inspectable, minimally privileged implementations. Document third-party license compliance and supply-chain risks.180181---182183## 4. NATIVE DESIGN SYSTEM: ORIX AESTHETIC184185Create a polished, original visual identity:186187- Premium graphite/dark surfaces with restrained electric violet/blue accents; support light appearance and system appearance.188- Strong hierarchy, excellent typography, precise alignment, generous whitespace, native vibrancy where appropriate.189- Sophisticated transitions and spring animations, respecting Reduce Motion.190- Custom iconography consistent with SF Symbols and Apple HIG.191- Beautiful empty states, loading states, skeletons, inline validation, and error recovery.192- Smooth workflow canvas with zoom, pan, minimap, snapping, selection, multi-select, keyboard navigation, grouping, and undo/redo.193- Meaningful live indicators for running, paused, waiting approval, failed, succeeded, canceled, and recovering.194- Inspector panels with input/output schemas, tool permissions, runtime metrics, and last-run data.195- Native searchable command palette and global keyboard shortcuts.196- User-configurable sidebar, resizable panels, multiple windows, and workspace persistence.197- Responsive behavior across compact laptop displays and large external monitors.198- Dynamic Type/accessibility accommodations, VoiceOver labels, high contrast, full keyboard navigation.199- Performance budgets for animation, list virtualization, logs, and graph rendering.200201### Primary application sections202203**Dashboard:** active runs, success/failure history, recent workflows, provider status, cost and token usage, scheduled jobs, attention-required approvals.204205**Agent Chat:** model selector, provider icon, streaming reasoning summaries (not hidden private chain-of-thought), tool activity, approvals, artifacts, file references, execution timeline, stop/retry buttons.206207**Workflow Studio:** visual DAG and controlled cycles, conditionals, loops, fan-out/fan-in, manual gates, retries, variables, triggers, tool nodes, agent nodes, error handlers.208209**Code Studio:** syntax highlighting, completion, validation, docs, diff viewer, formatter, snippets, test runner, terminal console, debugger, source control integration.210211**Agents:** reusable agents with instructions, provider/model, budgets, tool grants, memory scope, concurrency, and evaluation profiles.212213**Skills:** built-in and imported tool packages, capability manifest, versioning, signatures/trust metadata, install/remove, permissions, and local testing.214215**Runs:** searchable execution history with inputs, outputs, tool calls, screenshots (if consented), logs, cost, timing, error chains, replay, and export.216217**Scheduler:** calendar/timeline, file watchers, login triggers, wake/restart behavior, next run, missed runs, concurrency limits.218219**Settings:** providers, Keychain-backed credentials, permissions, privacy, appearance, storage, backup, telemetry opt-in, notifications, update channel, diagnostics.220221---222223## 5. AGENTIC ENGINE: THINK → ACT → OBSERVE → VERIFY224225Build a provider-independent orchestration engine that can:2262271. Parse the user objective and identify needed resources.2282. Inventory available tools and permission scopes.2293. Produce a plan with checkpoints and expected artifacts.2304. Execute tool calls with typed arguments and validated results.2315. Observe actual machine state and tool outputs.2326. Update the plan as new evidence arrives.2337. Recover from transient errors with bounded retries.2348. Ask for clarification only when ambiguity materially affects safety or correctness.2359. Escalate approval before consequential operations.23610. Verify postconditions before reporting success.23711. Save reusable knowledge and workflow improvements without leaking secrets.23812. Produce a final execution receipt with evidence and limitations.239240### Runtime primitives241242Implement `AgentSession`, `AgentTask`, `AgentPlan`, `ToolCall`, `ToolResult`, `ExecutionContext`, `PermissionGrant`, `ApprovalRequest`, `Artifact`, `Observation`, `VerificationResult`, `MemoryRecord`, and `RunReceipt` as typed, versioned data models.243244An agent loop should enforce:245246- Maximum turns, wall-clock time, tokens, monetary cost, tool-call count, subprocess duration, and concurrent tasks.247- Cancellation and interruption propagation to child tasks/processes.248- Explicit state machine transitions.249- Deterministic persistence at important boundaries.250- Tool idempotency keys where possible.251- Durable checkpoints and crash recovery.252- Output schema validation and refusal of malformed tool calls.253- Loop detection, repetitive-action detection, and budget exhaustion handling.254- Untrusted content separation: web pages, emails, PDFs, repository files, and tool output are **data**, not new system instructions.255- No hidden expansion of privileges based on model-generated text.256257### Planning modes258259- **Quick Action:** one-off task with immediate feedback.260- **Guided:** plan preview, approvals, execution, verification.261- **Autonomous:** agent can plan and retry within pre-approved scoped capabilities.262- **Scheduled:** persisted workflow executes on a trigger.263- **Review:** dry-run or read-only analysis with no side effects.264265Autonomous mode does not mean unrestricted root access or silent destructive behavior.266267---268269## 6. MULTI-PROVIDER AI ADAPTERS270271Build a clean provider interface with separate adapters for:272273### Anthropic274- Model discovery or maintained verified model registry.275- Messages API, streaming, tool use, structured output support as available.276- Token accounting, provider errors, backoff, and model-specific capabilities.277- Prompt caching when supported and appropriate.278279### OpenAI280- Current supported API endpoints, streaming, tools, structured outputs, and capability discovery.281- Explicit handling of model differences and unsupported features.282- Usage accounting and resilient error handling.283284### OpenRouter285- Model catalog retrieval, provider routing choices, model capability metadata, pricing when supplied.286- Compatibility mapping for tools, structured outputs, and streaming.287- Provider-specific limits and error reporting.288289### BYOK UX290291- Onboarding asks user to select provider and add key.292- Secrets stored only in macOS Keychain, scoped to the local user.293- Test connection with a minimal, explicit request; show cost warning if appropriate.294- Provider availability and latency indicators.295- Separate models for planner, executor, verifier, and specialist agents.296- Configurable max spend per task/day, token ceilings, concurrency, and retries.297- Never display full keys after entry; support rotate, replace, and delete.298- No hardcoded test credentials in binary, test data, or git history.299300### SP Chat development testing integration301302Inspect SP Chat deployment metadata and identify its provider credential injection pattern. Create an **optional developer-only** test adapter that references an approved existing secret source without extracting or duplicating values. Test the three providers independently with low-token, low-cost requests and explicit budget caps. If access is unavailable, use recorded sanitized fixtures and document that live provider verification is pending. Never ship the development adapter enabled in release builds.303304---305306## 7. TOOL REGISTRY AND EXECUTION CONTRACT307308Every tool must expose:309310```swift311protocol ORIXTool: Sendable {312 var identifier: String { get }313 var manifest: ToolManifest { get }314 func execute(315 input: ToolInput,316 context: ToolExecutionContext317 ) async throws -> ToolResult318}319```320321The manifest should include:322323- Human-readable description and version.324- JSON-compatible input/output schemas.325- Side-effect classification: read, write, external communication, destructive, privileged.326- Required macOS permissions and file/app scopes.327- Allowed working directories and network destinations.328- Default timeout, resource limits, and concurrency policy.329- Approval policy and dry-run support.330- Redaction rules for inputs/outputs.331- Test fixtures and verification hooks.332333Tools must never treat the model's requested action as permission in itself.334335### Built-in tool families336337**Filesystem:** list, search, read, write, copy, move, rename, metadata, checksums, compare, watch, archive, and restore. Trash by default for user-facing delete operations; confirm irreversible deletion.338339**Shell:** run allowlisted/scoped commands or user-approved scripts with explicit environment, working directory, process lifetime, stdout/stderr limits, and approval gates.340341**AppleScript/JXA:** discover scripting dictionaries, execute scripts, inspect results, report Apple Events authorization errors, and support reusable app adapters.342343**Accessibility:** inspect accessibility tree, locate controls, click, type, select, scroll, focus, and wait for state transitions after user authorization.344345**Screen/Vision:** capture authorized windows or regions, identify visual elements, compare before/after, annotate screenshots, and request user confirmation when uncertain.346347**Browser:** navigate permitted sites, inspect accessible page structure, interact with forms, download artifacts, and validate navigation. Prefer supported automation interfaces over brittle pixel clicking.348349**Python:** execute managed Python scripts with dependency manifests, isolated environments, package allow/deny policies, resource quotas, and saved artifacts.350351**Swift/JavaScript:** run explicitly scoped scripts with validation, timeouts, and capability controls; avoid unsafe in-process execution of untrusted code.352353**Documents:** read and transform text, CSV, JSON, Markdown, spreadsheets, images, and PDFs using local libraries where feasible.354355**Git:** status, diff, branches, commits, tests, and patch creation; require confirmation before push, force operations, or destructive history rewrites.356357**System:** app launch/quit, notifications, clipboard with explicit consent, window management, process status, power/network state observation.358359**HTTP:** typed requests with explicit domain allowlists and credentials isolated from prompts and logs.360361**Notifications:** local alerts, actionable approval notifications, run completion summaries.362363For each family, implement real tools with real integration tests and explicit capability declarations. Unsupported operations must return honest structured errors.364365---366367## 8. MACOS COMPUTER CONTROL368369Implement a hybrid control strategy with ordered fallback:3703711. Native application API or documented integration.3722. Apple Events / AppleScript / JXA when app supports scripting.3733. Accessibility API with semantic element identification.3744. Vision-assisted UI interaction only with explicit screen access and verification.3755. Keyboard/mouse emulation as a last resort, with focus and state checks.376377### Essential engineering details378379- Correctly detect frontmost application and focused window.380- Track display scaling, multi-monitor layouts, window movement, and coordinate transforms.381- Avoid clicking based on stale screenshots.382- Inspect accessible labels, roles, and values before action.383- Wait for observable postconditions rather than arbitrary sleeps.384- Recover from dialogs, permission prompts, unexpected windows, and application restarts.385- Avoid recording sensitive windows, passwords, payment forms, and secure text fields.386- Expose a visible screen-control indicator and emergency stop shortcut.387- Support a pause button that stops the agent before the next side-effecting action.388- Never attempt to automate macOS privacy consent prompts by bypassing user interaction.389- Explain limitations of apps that do not expose reliable accessibility or scripting interfaces.390391### Application integration candidates392393Finder, Safari, Mail, Calendar, Notes, Reminders, Preview, Numbers, Pages, Keynote, Terminal, Xcode, browsers, and user-installed third-party apps. Verify actual capabilities per app/version; do not claim universal compatibility.394395---396397## 9. WORKFLOW ENGINE AND VISUAL STUDIO398399Use a versioned, serializable workflow intermediate representation (IR).400401### Node categories402403- Trigger: manual, schedule, file event, app event, webhook/local HTTP listener if explicitly enabled, system event.404- Agent: planner, worker, specialist, verifier, reviewer.405- Tool: shell, AppleScript, Python, browser, file, app, document, HTTP.406- Logic: condition, switch, loop, foreach, map, reduce, filter, transform.407- Parallelism: fork, join, concurrency limiter, queue.408- Data: variable, constant, secret reference, file, JSON parser, schema validator.409- Control: approval, delay, retry, timeout, pause, cancel, fallback.410- Output: notification, document, artifact, log, report, clipboard with consent.411- Error: catch, compensate, rollback where possible, alert, escalate.412413### Workflow features414415- Typed ports and connection validation.416- Schema-aware autocompletion.417- Editable expressions with safe evaluation semantics.418- Named variables and secrets by reference only.419- Subworkflows, templates, reusable components.420- Version history, draft/published states, diffs, rollback.421- Import/export with validation and secret stripping.422- Dry-run with explicit indication of what is simulated.423- Step-by-step execution and breakpoints.424- Retry policy per node, backoff, timeout, and error routes.425- Compensation actions where reversible; never promise rollback for irreversible external actions.426- Live node highlighting, progress, timing, and result inspection.427- Scheduling and dependency management.428- Local template gallery.429430### Example declarative workflow (illustrative; define and validate actual grammar)431432```yaml433apiVersion: orix.ai/v1434kind: Automation435metadata:436 name: weekly-research-report437spec:438 trigger:439 type: schedule440 cron: "0 9 * * MON"441 timezone: America/Toronto442 permissions:443 filesystem:444 read: ["~/Documents/Research"]445 write: ["~/Documents/Reports"]446 processes: ["python"]447 budget:448 maxDurationSeconds: 1800449 maxProviderCostUSD: 2.00450 steps:451 - id: discover452 tool: filesystem.list453 with:454 path: "~/Documents/Research"455 - id: analyze456 agent: research-analyst457 needs: [discover]458 - id: render459 tool: document.renderPDF460 needs: [analyze]461 - id: verify462 verifier: artifact-exists463 needs: [render]464```465466Do not assume `~` expansion, cron syntax, or paths are safe without implementing explicit resolution and validation.467468---469470## 10. CODE STUDIO AND SCRIPT AUTHORING471472Provide an advanced native IDE-like experience:473474- Editor with syntax highlighting for YAML/JSON, AppleScript, Bash, Python, Swift, JavaScript.475- Language-aware validation and formatting.476- Search/replace, command palette, multi-file project view, diagnostics.477- Integrated terminal with explicit sandbox/scoping and visible process controls.478- Script templates, snippets, autocomplete, inline tool docs.479- Run selected script, test script, inspect stdout/stderr, save as skill.480- Test fixtures, mock tool responses, and live test mode.481- Visual diff and approval of agent-generated changes.482- Version control integration and reproducible environment manifests.483- Inline AI assistant that can explain, generate, refactor, and test automation scripts.484485Do not confuse a raw terminal with an unrestricted privilege grant. The terminal must clearly distinguish user-initiated shell execution from agent-initiated shell execution and apply the corresponding security policy.486487---488489## 11. WATCH & LEARN (ADVANCED MILESTONE)490491Create an opt-in recording mode that observes an explicitly chosen workflow:4924931. User presses Start Recording.4942. App requests required permissions through system flows.4953. Show persistent recording indicator and pause/stop controls.4964. Record only necessary UI events and selected application context.4975. Mask sensitive fields and allow redaction before saving.4986. Infer reusable semantic actions rather than hardcoding coordinates.4997. Ask the user to label variables and parameterize inputs.5008. Generate a proposed workflow and show its permissions.5019. Test in a controlled environment.50210. Save only after user approval.503504Never capture credentials, private messages, or unrelated windows silently. Recording should be off by default.505506---507508## 12. AUTONOMOUS SKILL CREATION509510An agent may propose a new reusable tool when no existing tool suffices.511512Pipeline:5135141. Identify missing capability and desired contract.5152. Generate a manifest with input/output schemas and requested permissions.5163. Generate implementation in a supported runtime.5174. Static analysis, dependency inspection, and security review.5185. Run tests in an isolated environment with synthetic fixtures.5196. Show a source diff and capability summary to the user.5207. Request approval for installation and runtime grants.5218. Install as a versioned local skill.5229. Execute with bounded permissions.52310. Monitor failures and support rollback to prior versions.524525Support skill signing/checksums, provenance, semantic versioning, disable/uninstall, and dependency pinning. Imported skill instructions and documentation are untrusted content. They must not override system security policy.526527---528529## 13. SECURITY MODEL: POWER WITHOUT UNCONTROLLED AUTHORITY530531Treat this as a first-class engineering deliverable, not a disclaimer.532533### Capability permissions534535Represent permissions as specific grants, such as:536537- Read a user-selected directory.538- Write a user-selected directory.539- Automate a named application.540- Read a selected browser page.541- Capture a selected window.542- Execute a specific script/runtime.543- Connect to specific network domains.544- Access a specific Keychain secret reference for a specific provider client.545- Send external communications.546- Modify or delete files.547548Grants must be scoped, reviewable, revocable, and recorded. Persistent grants should be opt-in and clearly differentiated from one-time approval.549550### Approval tiers551552**Tier 0 — Read-only:** scoped local inspection and analysis.553554**Tier 1 — Reversible local write:** user-approved workspace modifications with backup or undo where feasible.555556**Tier 2 — Consequential:** external sends, Git pushes, installs, app changes, sensitive file writes; explicit approval or narrowly preauthorized workflow policy.557558**Tier 3 — High-risk/irreversible:** permanent deletion, privileged system changes, financial transfers, credential modifications, destructive Git operations; explicit contemporaneous user approval and additional safeguards. Some operations should remain unsupported.559560### Security controls561562- Enforce permissions in the execution layer, not in prompts alone.563- Use a policy engine with deny-by-default capability checks.564- Never execute arbitrary model-authored shell text solely because it appeared in a model response.565- Treat command arguments as structured values; avoid shell interpolation when possible.566- Redact tokens, passwords, private file contents, and PII from logs and telemetry.567- Prevent prompt injection from files, websites, tool output, and retrieved documents.568- Prevent secret exfiltration through tool chaining or external network calls.569- Prevent privilege escalation through helper processes.570- Validate XPC caller identity and enforce least privilege.571- Keep helper entitlements minimal and auditable.572- Add rate limits, quotas, and watchdogs.573- Require explicit user action for OS permission grants.574- Support emergency stop, revoke-all, and kill active subprocess trees.575- Make privacy/telemetry opt-in; do not send screen captures to a provider without clear user consent.576- Encrypt sensitive persisted data where needed and use Keychain for secrets.577- Maintain an append-only or tamper-evident local audit trail for consequential actions where practical.578- Audit third-party dependencies and ensure signed release artifacts.579580### Threat-model scenarios to test5815821. Malicious web page instructs agent to upload local secrets.5832. PDF contains fake system instructions.5843. Skill package requests broader access than its manifest declares.5854. Tool output includes shell injection payload.5865. Agent repeatedly retries an irreversible action.5876. Model invents success after tool failure.5887. Provider key accidentally appears in console output.5898. Browser focus shifts before a click.5909. Background workflow resumes after reboot with stale permission.59110. Remote service returns hostile content that attempts privilege escalation.592593Document mitigations and residual risks.594595---596597## 14. MEMORY, CONTEXT, AND KNOWLEDGE598599Implement a durable, local-first memory architecture:600601- Session memory for the current task.602- Workflow memory for stable run parameters and learned fixes.603- User-approved preferences and reusable conventions.604- Skill knowledge, manifests, examples, and tool documentation.605- Execution memory with verified observations and artifact pointers.606- Retrieval over locally indexed approved data; scope access per workflow.607- Summarization/compaction to manage context windows.608- Provenance and timestamps for remembered facts.609- Explicit delete/export controls.610- No silent long-term retention of sensitive screen content.611- No persistence of secrets in prompts, memory summaries, or embeddings.612613Memory is advisory, not authorization. A remembered approval does not grant a new capability.614615---616617## 15. SCHEDULER AND ALWAYS-ON SERVICE618619Build a robust background execution system using appropriate macOS launchd/SMAppService patterns and explicit user opt-in.620621Support:622623- Manual runs.624- One-time schedules.625- Recurring schedules with timezone and daylight-saving behavior.626- File/directory changes with debouncing.627- Login/startup triggers.628- Application lifecycle events where supported.629- Network/power state triggers where feasible.630- Missed-run policy after sleep/restart.631- Durable queue, retry, deduplication, idempotency.632- Concurrent run limits and per-workflow locks.633- Pause/resume, cancel, run now, disable.634- Visible menu-bar status and notifications.635- Approval-required state that waits safely for user interaction.636637Do not promise execution while the Mac is powered off. Handle sleep, locked sessions, and GUI automation restrictions honestly: headless background jobs can run in more situations than UI automation.638639---640641## 16. OBSERVABILITY, COSTS, AND EVIDENCE642643Every run should expose:644645- Run ID, workflow version, start/end times, status.646- Agent/provider/model choices.647- Tool calls, sanitized arguments, results, duration.648- Token usage and estimated/actual provider costs where available.649- Error classification and retry history.650- Permissions used and approvals granted.651- Produced artifacts and hashes when appropriate.652- Verification assertions and outcomes.653- Resource usage, queue time, and concurrency.654- Screenshots only when explicitly authorized.655- Final machine-readable and human-readable execution receipt.656657### Outcome statuses658659`SUCCEEDED`, `PARTIALLY_SUCCEEDED`, `FAILED`, `CANCELED`, `TIMED_OUT`, `WAITING_APPROVAL`, `BLOCKED_PERMISSION`, `BUDGET_EXCEEDED`, `RECOVERY_REQUIRED`.660661Never mark success solely because the language model says it succeeded. Verify the relevant postcondition.662663---664665## 17. ERROR RECOVERY AND RESILIENCE666667- Typed errors for provider, tool, permission, OS, network, parsing, timeout, and validation failures.668- Bounded exponential backoff with jitter for transient network errors.669- No blind retries for non-idempotent actions.670- Workflow checkpointing and restart recovery.671- Human-readable error explanations with actionable fixes.672- Automatic recovery only within the original permission scope.673- Compensation/undo for supported reversible operations.674- Safe cancellation of subprocess groups.675- Disk-space and resource-pressure monitoring.676- Schema migration and corrupt database recovery.677- Backup/export/import for workflows and user data.678- Offline mode for local tools, with provider-dependent features clearly marked unavailable.679680---681682## 18. BUILT-IN AUTOMATIONS TO SHIP683684Ship tested, editable examples:6856861. Smart Downloads Organizer.6872. PDF to structured data and report.6883. Weekly research digest from selected local files.6894. Git repository test-and-review assistant.6905. CSV cleanup and chart generation.6916. Batch image conversion and naming.6927. Calendar-based preparation workflow (where integration is permitted).6938. Folder backup and verification.6949. Website change monitoring using approved access.69510. Local system health report.69611. Meeting notes organization from user-provided transcripts.69712. Document metadata extraction and cataloging.69813. Scheduled screenshot of a selected app/window, with consent.69914. Automated spreadsheet validation.70015. Multi-step browser data-entry workflow on a test site.701702Each example needs: description, permissions, configurable parameters, sample data, tests, expected artifacts, and explicit limitations.703704---705706## 19. QUALITY ASSURANCE AND TESTING707708Create an automated test pyramid:709710### Unit tests711- Workflow schema validation and migrations.712- Graph execution order, cycles, branches, and retries.713- Provider response parsing and stream handling.714- Tool manifest validation.715- Permission policy decisions.716- Budget enforcement and cancellation.717- Secret redaction and log filtering.718- Scheduler timezone and DST behavior.719- Persistence and crash recovery.720721### Integration tests722- Real filesystem operations inside temporary directories.723- Real subprocess execution and cancellation.724- AppleScript interaction with a safe test app or supported built-in app.725- Accessibility tree inspection under granted permission.726- Provider smoke tests using approved SP Chat-backed developer credentials.727- Local browser automation against a controlled test page.728- Keychain create/read/delete using test-only entries.729- Background agent communication and lifecycle.730731### End-to-end tests732- Onboard with a test provider.733- Create workflow from natural language.734- Review permissions and approve.735- Run and inspect live tool events.736- Generate and verify an artifact.737- Save, reopen, modify, and rerun workflow.738- Schedule and observe a background run.739- Cancel a running job and verify subprocess cleanup.740- Recover from simulated provider and tool failures.741- Export/import workflow without secrets.742743### Security tests744- Prompt injection attempts.745- Path traversal and symlink escape.746- Command injection.747- Credential redaction.748- Unauthorized tool escalation.749- Excessive token/tool loops.750- Unexpected GUI focus change.751- Malicious skill manifests.752- Non-idempotent retry protection.753754### Performance tests755- Cold launch and warm launch.756- Memory footprint while idle and running.757- Large workflow graph rendering.758- Large log streaming and virtualization.759- Long-running background jobs.760- UI responsiveness during provider streaming and subprocess output.761762Do not substitute mocked tests for live integrations. Distinguish mocked, integration, and verified-on-device results in test reports.763764---765766## 20. APPLE SIGNING, NOTARIZATION, AND DISTRIBUTION767768The development Mac is expected to already contain signing/notarization configuration. Discover and reuse it securely.769770### Release pipeline7717721. Validate Xcode scheme, bundle ID, version/build numbers, minimum macOS target, and entitlements.7732. Build Release configuration for intended Apple Silicon and, if feasible, Intel architectures.7743. Run all automated tests and static checks.7754. Verify no API keys, tokens, internal hostnames, private test fixtures, or development secrets are embedded.7765. Sign app and bundled helpers with the appropriate Developer ID identity and hardened runtime.7776. Validate entitlements and code signatures using `codesign` and `spctl`.7787. Package as a signed installer or DMG as appropriate.7798. Submit to Apple notarization via `xcrun notarytool` using an existing authorized keychain profile.7809. Wait for and inspect Apple's notarization result; handle rejection logs.78110. Staple the notarization ticket where supported.78211. Validate Gatekeeper acceptance on the finished artifact.78312. Generate checksums, release notes, version metadata, and installation instructions.78413. Test first launch, permissions onboarding, updates, helper registration, and uninstall on a clean Mac/user profile where possible.78514. Store non-sensitive release evidence in `docs/release/`.786787### Strict release requirements788789- No secret export from Keychain.790- No embedding SP Chat credentials in distributed binaries.791- No false claim of notarization before Apple accepts the submission.792- No silent disabling of hardened runtime or Gatekeeper.793- No entitlement escalation without explicit technical justification.794- No claim of Mac App Store compliance without an actual policy and entitlement review.795- Never publish a release without verifying the final artifact.796797Investigate a secure update mechanism (for example, a well-maintained signed updater) and verify its security model. Do not implement insecure self-update downloads.798799---800801## 21. DOCUMENTATION REQUIREMENTS802803Produce comprehensive, accurate documentation:804805- Product README with screenshots from the actual running app.806- Installation and first-run onboarding.807- Provider setup for Anthropic, OpenAI, OpenRouter.808- Explanation of local BYOK security.809- Permissions and privacy guide.810- Workflow language reference and JSON/YAML schema.811- Tool API and capability manifest specification.812- AppleScript, JXA, Shell, Python, and browser automation guides.813- Skill development tutorial and security review checklist.814- Visual editor tutorial.815- Scheduling and background operation guide.816- Troubleshooting common TCC/Accessibility/Apple Events issues.817- Threat model and security architecture.818- Backup, migration, and recovery guide.819- Release and notarization runbook.820- Changelog, known limitations, roadmap.821822Document actual implementation behavior, not aspirational features.823824---825826## 22. MILESTONE PLAN — IMPLEMENT IN ORDER827828### M0 — Discovery, architecture, security foundations829830Deliver environment audit, provider credential strategy, signing/notary discovery, threat model, repository setup, project architecture, CI/build scripts, test harness, design tokens.831832**Gate:** project builds cleanly and security boundaries are documented.833834### M1 — Native app foundation835836Deliver window system, navigation, dashboard, design system, onboarding, settings, Keychain, local database, run/event models, console UI, permission center.837838**Gate:** native app launches and persists user settings; Keychain tests pass.839840### M2 — Providers and agent loop841842Deliver Anthropic/OpenAI/OpenRouter adapters, model selection, streaming chat, tool-calling protocol, execution state machine, budget controls, cancellation, approval UI.843844**Gate:** provider fixtures pass and at least one authorized live smoke test per accessible provider succeeds; real tool calls work.845846### M3 — Core macOS tools847848Deliver filesystem, shell, AppleScript/JXA, app control, Accessibility integration, local browser interaction, Python execution, artifact verification.849850**Gate:** end-to-end tasks execute on the Mac with real postcondition checks.851852### M4 — Workflow studio and code studio853854Deliver versioned workflow IR, graph canvas, visual editing, declarative editor, validation, step debugger, live execution visualization, templates.855856**Gate:** create/edit/save/reopen/run workflow across modes without losing semantics.857858### M5 — Background scheduling and reliability859860Deliver launch agent/service, scheduled jobs, file watchers, durable queue, resume policy, notification center, retry/recovery, export/import.861862**Gate:** scheduled workflow survives app-window closure and handles sleep/restart appropriately.863864### M6 — Multi-agent and self-extending skills865866Deliver agent orchestration, specialist delegation, scoped memory, skill SDK, skill generation pipeline, source review, sandboxed test harness, versioned installs.867868**Gate:** an agent can propose, test, approve, install, and run a new skill safely.869870### M7 — Watch & Learn and advanced polish871872Deliver consent-based observation, workflow inference, parameterization, sophisticated animations, accessibility audit, performance optimization.873874**Gate:** recorded workflow is replayable with user approval and sensitive information is protected.875876### M8 — Production hardening and release877878Deliver full regression tests, security review, documentation, signed release, Apple notarization, stapling, Gatekeeper verification, DMG/installer, release notes.879880**Gate:** reproducible, verified distribution artifact; report any Apple/account blockers honestly.881882Do not prematurely call the product complete if advanced milestones remain unimplemented. Deliver incremental usable builds.883884---885886## 23. DEFINITION OF DONE887888The project is done only when:889890- A genuine native SwiftUI macOS app exists and builds.891- Onboarding, provider selection, and Keychain-backed BYOK work.892- Anthropic, OpenAI, and OpenRouter integrations are implemented and tested at the appropriate level.893- The agent can plan, call real tools, observe, and verify results.894- macOS automation uses supported APIs and respects consent.895- Users can create and run workflows in natural language, visual, and code views.896- The scheduler and background worker execute reliable persisted jobs.897- The console shows real logs, costs, tool calls, errors, and artifacts.898- Dangerous operations are gated by enforceable permission checks.899- Skills are installable, testable, versioned, and permission-scoped.900- Built-in examples work with actual fixtures and clear setup instructions.901- Automated tests and at least one full end-to-end Mac execution pass.902- Documentation is accurate and includes screenshots of the real app.903- Release builds do not contain development credentials.904- Signing and notarization have been genuinely completed or their blockers explicitly reported.905906---907908## 24. ENGINEERING DISCIPLINE AND AUTONOMOUS EXECUTION909910At every significant step:9119121. Inspect existing code and dependencies.9132. State a small implementation objective.9143. Make coherent changes.9154. Build and run relevant tests.9165. Investigate failures rather than hiding them.9176. Update documentation and architecture decisions.9187. Commit logical milestones if repository policy allows.9198. Continue to the next independently achievable milestone.920921Maintain `docs/implementation-status.md` with:922923- Completed and verified capabilities.924- In-progress work.925- Known defects and severity.926- Required human approvals.927- Live integration test status by provider.928- Security test outcomes.929- Release readiness and notarization status.930931Do not pad progress with empty files, cosmetic demos, or unimplemented buttons. A button that claims to run, save, stop, retry, or export must perform that action. UI states must be connected to actual runtime state.932933### Final deliverables9349351. Complete native macOS source project.9362. Real functional app and background helper.9373. Provider integrations and safe developer test configuration.9384. Visual workflow editor and code IDE.9395. Tool/skill SDK and built-in automations.9406. Test suite and security audit results.9417. Full documentation and screenshots.9428. Signed/notarized release artifact when authorized configuration permits.9439. Concise final engineering report: features, tests, known limitations, security decisions, distribution status, and next steps.944945---946947## 25. START NOW948949**Begin by auditing the current repository, local macOS/Xcode environment, SP Chat deployment's authorized provider test configuration, and existing Apple signing/notarization metadata. Then create the implementation plan and immediately build the first working native application milestone.**950951Do not ask the product owner to repeat information already discoverable in authorized local configuration. Ask only when permission, an external credential authorization, a consequential action, or an architectural decision truly requires a human choice. Never mistake discoverability for permission to use a secret or perform an irreversible action.952953**ORIX OS Automation should feel like a powerful, trustworthy extension of macOS: the user defines the goal, the system carries out authorized work, and every result can be inspected and verified.**954955<!-- spb-source-governance:start -->956## Source header policy (SP Boucher)957958When creating or modifying eligible first-party source files, follow the SP Boucher Source Header959Policy. Preserve existing copyright and license notices, select the correct language-specific960template, and avoid adding identifying information to anonymous academic submission artifacts.961962- Repository policy: `.spb-source-governance.toml` (project, contact, verified rights, license).963- Add or refresh headers with `spb-header apply --repo .`; verify with `spb-header check --repo .`.964- Never hand-write a header, never add one to generated, vendored, third-party or data files.965<!-- spb-source-governance:end -->966