SPB Git

Repository Public

README.md 41.8 KB 709 lines markdown

1<div align="center">23<img src="docs/brand/logo.svg" alt="SP Notes Core logo" width="560">45<br><br>67<img src="docs/brand/icon.png" alt="SP Notes Core icon" width="112" height="112">89# SP Notes Core1011**The shared, UI-free Swift core of the native SP Notes apps for iOS, iPadOS and macOS — a lossless ProseMirror document model, a durable local SQLite store, a typed API client with passkeys, and a synchronisation engine whose first rule is that no local modification is ever lost.**1213**Author: [Simon-Pierre Boucher](https://www.spboucher.ai)**1415[![backend](https://img.shields.io/website?url=https%3A%2F%2Fnote.spboucher.ai%2Fapi%2Fhealth&label=backend%20note.spboucher.ai&style=for-the-badge&up_message=online&down_message=offline)](https://note.spboucher.ai)16![version](https://img.shields.io/badge/version-2.0.0-3157D5?style=for-the-badge)17![platforms](https://img.shields.io/badge/platforms-iOS%2018%2B%20%C2%B7%20macOS%2015%2B-000000?style=for-the-badge&logo=apple&logoColor=white)18![license](https://img.shields.io/badge/license-all%20rights%20reserved-lightgrey?style=for-the-badge)19![author](https://img.shields.io/badge/author-Simon--Pierre%20Boucher-0A66C2?style=for-the-badge)2021![Swift](https://img.shields.io/badge/Swift-6%20language%20mode-F05138?style=flat-square&logo=swift&logoColor=white)22![swift-tools-version](https://img.shields.io/badge/swift--tools--version-6.0-F05138?style=flat-square&logo=swift&logoColor=white)23![Swift Package Manager](https://img.shields.io/badge/SwiftPM-5%20library%20products-F05138?style=flat-square)24![dependencies](https://img.shields.io/badge/third--party%20dependencies-0-brightgreen?style=flat-square)25![SQLite](https://img.shields.io/badge/SQLite-system%20%C2%B7%20WAL%20%C2%B7%20FTS5-003B57?style=flat-square&logo=sqlite&logoColor=white)26![AuthenticationServices](https://img.shields.io/badge/AuthenticationServices-passkeys-3423A6?style=flat-square&logo=webauthn&logoColor=white)27![Keychain](https://img.shields.io/badge/Security-Keychain%20token%20store-555?style=flat-square)28![concurrency](https://img.shields.io/badge/concurrency-actors%20%C2%B7%20Sendable-informational?style=flat-square)29![Swift Testing](https://img.shields.io/badge/Swift%20Testing-%40Test-F05138?style=flat-square)30![XCTest](https://img.shields.io/badge/XCTest-API%20module-informational?style=flat-square)3132[![commits](https://git.spboucher.ai/badge/sp-notes-core/commits.svg?style=flat-square)](https://git.spboucher.ai/sp-notes-core/commits)33[![last commit](https://git.spboucher.ai/badge/sp-notes-core/last-commit.svg?style=flat-square)](https://git.spboucher.ai/sp-notes-core/commits)34[![activity](https://git.spboucher.ai/badge/sp-notes-core/activity.svg?style=flat-square)](https://git.spboucher.ai/sp-notes-core/insights)35[![language](https://git.spboucher.ai/badge/sp-notes-core/language.svg?style=flat-square)](https://git.spboucher.ai/sp-notes-core)36[![languages](https://git.spboucher.ai/badge/sp-notes-core/languages.svg?style=flat-square)](https://git.spboucher.ai/sp-notes-core/insights)37[![size](https://git.spboucher.ai/badge/sp-notes-core/size.svg?style=flat-square)](https://git.spboucher.ai/sp-notes-core)38[![branches](https://git.spboucher.ai/badge/sp-notes-core/branches.svg?style=flat-square)](https://git.spboucher.ai/sp-notes-core/branches)39[![tags](https://git.spboucher.ai/badge/sp-notes-core/tags.svg?style=flat-square)](https://git.spboucher.ai/sp-notes-core/tags)40[![clones](https://git.spboucher.ai/badge/sp-notes-core/clones.svg?style=flat-square)](https://git.spboucher.ai/sp-notes-core)4142![modules](https://img.shields.io/badge/modules-4%20%2B%20umbrella-blue?style=flat-square)43![source files](https://img.shields.io/badge/source%20files-39%20Swift-blue?style=flat-square)44![source lines](https://img.shields.io/badge/source%20lines-10%20936-blue?style=flat-square)45![test files](https://img.shields.io/badge/test%20files-27%20Swift-informational?style=flat-square)46![test lines](https://img.shields.io/badge/test%20lines-9%20054-informational?style=flat-square)47![tests](https://img.shields.io/badge/tests-400-brightgreen?style=flat-square)48![node types](https://img.shields.io/badge/schema%20node%20types-30-blue?style=flat-square)49![local tables](https://img.shields.io/badge/local%20tables-10%20%2B%20FTS5%20index-blue?style=flat-square)50![API methods](https://img.shields.io/badge/NotesAPI%20methods-47-blue?style=flat-square)51![docs](https://img.shields.io/badge/API%20documents-4%20%2B%20backend%20contract-informational?style=flat-square)5253![package](https://img.shields.io/badge/package-SPNotesCore-F5820B?style=flat-square)54![products](https://img.shields.io/badge/products-SPNotesDocument%20%C2%B7%20SPNotesStore%20%C2%B7%20SPNotesAPI%20%C2%B7%20SPNotesSync%20%C2%B7%20SPNotesCore-F5820B?style=flat-square)55![default localization](https://img.shields.io/badge/default%20localization-fr-informational?style=flat-square)56![consumers](https://img.shields.io/badge/consumers-SP%20Notes%20macOS%20%C2%B7%20SP%20Notes%20iOS-000000?style=flat-square&logo=apple&logoColor=white)57![isolated builds](https://img.shields.io/badge/isolated%20builds-SPN__ONLY%3DDocument%7CStore%7CAPI%7CSync-555?style=flat-square)5859</div>6061---6263## Table of contents6465- [Overview](#-overview)66- [Features](#-features)67- [Architecture](#-architecture)68- [Stack](#-stack)69- [Repository layout](#-repository-layout)70- [Getting started](#-getting-started)71- [Configuration](#-configuration)72- [Module: SPNotesDocument](#-module-spnotesdocument)73- [Module: SPNotesStore](#-module-spnotesstore)74- [Module: SPNotesAPI](#-module-spnotesapi)75- [Module: SPNotesSync](#-module-spnotessync)76- [Local data model](#-local-data-model)77- [Security and privacy](#-security-and-privacy)78- [Testing](#-testing)79- [Troubleshooting and gotchas](#-troubleshooting-and-gotchas)80- [Known limitations](#-known-limitations)81- [Documentation](#-documentation)82- [Author](#-author)83- [License](#-license)8485---8687## 🎯 Overview8889`SPNotesCore` is a Swift package with **no user interface and no third-party dependency**. It contains everything the90two native SP Notes apps have in common: how a note is represented, where it is stored on the device, how the app talks91to the backend, and how the device and the server are kept in step.9293The backend is [note-spboucher-ai](https://git.spboucher.ai/note-spboucher-ai) (production:94**https://note.spboucher.ai**), which stores a note as ProseMirror / Tiptap JSON and exposes a change feed for native95clients. The core must stay byte-compatible with it: plain text, word count and Markdown export reproduce the server's96own helpers, and every document the core emits is validated against the real web schema.9798Two rules run through the whole package:99100- **No note is ever lost** (store): local edits are never overwritten by the feed; every mutation is one atomic SQLite101  transaction; nothing waits for the network.102- **No local modification is lost or silently overwritten** (sync): concurrent edits are merged when that is safe;103  otherwise both versions are kept and the user decides.104105### Family106107| Repository | Role | Platform |108|---|---|---|109| [note-spboucher-ai](https://git.spboucher.ai/note-spboucher-ai) | Web app, API, database, sync feed | Web / PWA — note.spboucher.ai |110| **[sp-notes-core](https://git.spboucher.ai/sp-notes-core)** (this one) | Shared Swift package | iOS 18+ · macOS 15+ |111| [sp-notes-macos](https://git.spboucher.ai/sp-notes-macos) | SP Notes for Mac | macOS |112| [sp-notes-ios](https://git.spboucher.ai/sp-notes-ios) | SP Notes for iPhone and iPad | iOS / iPadOS |113114Both apps expect this repository to be cloned **next to them as `SPNotesCore`** (they reference `../SPNotesCore`).115116## ✨ Features117118| Module | What it gives |119|---|---|120| **SPNotesDocument** | ProseMirror JSON ⇄ a flat list of lines an editor can drive, without losing anything (unknown nodes included); editing commands (blocks, lists, marks, links, tables); plain text, word count, preview, Markdown import/export, HTML export; three-way merge by block id; schema validation |121| **SPNotesStore** | Durable local database on the system SQLite: pages, tags, attachments, ops queue, conflicts, local versions, reminders, meta; FTS5 search that ignores case and diacritics; change notification streams; multi-process access (app, share extension, read-only widget) |122| **SPNotesAPI** | The only module that talks to the backend: hardened `URLSession` transport, hand-written cookies, typed errors, one method per route, multipart uploads, downloads, server-sent events for the AI routes, passkey ceremonies through AuthenticationServices, Keychain token store |123| **SPNotesSync** | `SyncEngine` actor: push then pull in bounded rounds, idempotent retries, lost-acknowledgement recovery, three-way merge, conflict records, trash/purge/server-restore handling, background downloads of attachments |124| **SPNotesCore** | Umbrella module: one `import SPNotesCore` re-exports the four others |125126## 🏗 Architecture127128```mermaid129flowchart TB130    subgraph APPS["Native apps (own the UI and the attachment directory)"]131        MAC["SP Notes macOS<br/>AppKit · TextKit 2"]132        IOS["SP Notes iOS / iPadOS<br/>SwiftUI · UIKit · TextKit 2"]133    end134    subgraph CORE["SPNotesCore (this package)"]135        SYNC["SPNotesSync<br/>SyncEngine (actor)"]136        DOC["SPNotesDocument<br/>Foundation only · value types"]137        STORE["SPNotesStore<br/>NotesStore (actor) · SQLite WAL · FTS5"]138        API["SPNotesAPI<br/>HTTPClient (actor) · NotesAPI · passkeys"]139    end140    BACK["note-spboucher-ai<br/>https://note.spboucher.ai"]141    DB[("SQLite file<br/>on the device")]142    KC["Keychain<br/>session token"]143    MAC --> CORE144    IOS --> CORE145    SYNC --> DOC146    SYNC --> STORE147    SYNC --> API148    STORE --> DB149    API --> KC150    API -- "HTTPS · cookie written by hand" --> BACK151```152153The three lower modules are **independent of one another**:154155- `SPNotesStore` does not import `SPNotesDocument`: note content is opaque JSON `Data`, and the caller supplies the156  derived plain text, word count and preview.157- `SPNotesAPI` depends on Foundation, Security and AuthenticationServices only: note `content` and `properties` travel158  as `RawJSON` and are interpreted by `SPNotesDocument`.159- Only `SPNotesSync` depends on the three others.160161### One sync cycle162163```mermaid164sequenceDiagram165    autonumber166    participant A as App167    participant E as SyncEngine168    participant S as NotesStore169    participant B as Backend170    A->>S: local mutation (one SQLite transaction)171    A->>E: schedule() — debounced, 2 s by default172    loop up to 4 rounds173        E->>B: POST /api/pages (creations, parents first, client-chosen id)174        E->>B: POST /api/attachments (uploads)175        E->>B: PATCH with baseVersion (title · content · properties)176        alt 409 conflict177            E->>S: reconcile: adopt · move base · three-way merge · or record a conflict178        end179        E->>B: queued actions with their idempotency key, then reminders180        E->>B: GET /api/sync from the stored cursor, until more is false181        E->>S: apply each answer in one transaction with its cursor182    end183    E->>B: download up to 40 wanted files184    E-->>A: statusUpdates() — phase, pending, conflicts, needsReauth185```186187## 🧱 Stack188189| Layer | Technology | Version | Role |190|---|---|---|---|191| Language | Swift, language mode 6 on every target | tools 6.0 | Strict concurrency: actors, `Sendable` value types |192| Package | Swift Package Manager | — | 4 libraries + the umbrella; `defaultLocalization: "fr"` |193| Platforms | iOS, macOS | 18 · 15 (minimum) | Deployment targets of the package |194| Storage | system SQLite (`linkedLibrary("sqlite3")`) | — | WAL, `synchronous=FULL`, FTS5 |195| Network | Foundation `URLSession` (ephemeral) | — | TLS ≥ 1.2, no cookie storage, no cache, redirects refused |196| Sign-in | AuthenticationServices | — | Platform passkeys |197| Secrets | Security (Keychain) | — | Session token, this device only, not synchronised |198| Tests | Swift Testing (`@Test`) and XCTest | — | 301 + 99 tests |199| Third-party code | none | — | — |200201## 🗂 Repository layout20220387 tracked files.204205```206sp-notes-core/                       # clone it as SPNotesCore next to the apps207├── Package.swift                    # products, SPN_ONLY switch208├── Sources/209│   ├── SPNotesDocument/             # 14 files · 3 467 lines210│   │   ├── Model.swift  JSONValue.swift  Schema.swift  SchemaValidation.swift  BlockIds.swift211│   │   ├── Commands.swift  InlineCommands.swift  Inline.swift  Tables.swift212│   │   ├── Text.swift  MarkdownImport.swift  MarkdownExport.swift  HTMLExport.swift213│   │   └── Merge.swift214│   ├── SPNotesStore/                # 9 files · 2 966 lines215│   │   ├── NotesStore.swift  SQLite.swift  Schema.swift  Models.swift216│   │   └── NotesStore+Pages / +Queries / +Sync / +Queue / +History .swift217│   ├── SPNotesAPI/                  # 14 files · 3 479 lines218│   │   ├── APIConfiguration.swift  HTTPClient.swift  APIError.swift  JSONSupport.swift219│   │   ├── NotesAPI.swift  PageModels.swift  ServiceModels.swift  AIModels.swift220│   │   ├── MultipartForm.swift  SSEParser.swift221│   │   └── AuthService.swift  PasskeyCeremony.swift  PasskeyAuthorizer.swift  SessionTokenStore.swift222│   ├── SPNotesSync/                 # 1 file · 1 014 lines223│   │   └── SyncEngine.swift224│   └── SPNotesCore/                 # umbrella: @_exported import of the four modules225├── Tests/226│   ├── SPNotesDocumentTests/        # 5 suites + Fixtures/ (6 JSON files)227│   ├── SPNotesStoreTests/           # 5 suites228│   ├── SPNotesAPITests/             # 6 suites (URLProtocol stub + live tests)229│   └── SPNotesSyncTests/            # 6 suites + FakeBackend.swift (in-memory backend, 1 072 lines)230└── docs/231    ├── DOCUMENT_API.md  STORE_API.md  API_CLIENT.md  SYNC_ENGINE.md232    ├── reference/                   # read-only copies of the web sources the core must match + BACKEND_CONTRACT.md233    └── brand/234```235236## 🚀 Getting started237238### Build and test239240```sh241swift build242swift test243```244245One module in isolation (parallel development — each with its own scratch directory):246247```sh248SPN_ONLY=Document swift test --scratch-path .build-document249SPN_ONLY=Store    swift test --scratch-path .build-store250SPN_ONLY=API      swift build --scratch-path .build-api && SPN_ONLY=API swift test --scratch-path .build-api251SPN_ONLY=Sync     swift test --scratch-path .build-sync252```253254`SPN_ONLY=Sync` also declares the three modules it depends on. The umbrella product `SPNotesCore` only exists when255`SPN_ONLY` is unset.256257### Use it from an app258259Clone this repository next to the app as `SPNotesCore` and reference it as a local package:260261```swift262.package(path: "../SPNotesCore")263// then depend on the product "SPNotesCore" (everything) or on a single module264```265266```swift267import SPNotesCore268269let configuration = APIConfiguration.production(appVersion: "1.0.0")      // https://note.spboucher.ai270let tokens = KeychainTokenStore(configuration: configuration, accessGroup: "<team>.<keychain group>")271let client = HTTPClient(configuration: configuration, tokenStore: tokens)272let store  = try NotesStore(url: databaseURL)273let engine = SyncEngine(store: store, api: NotesAPI(client: client), files: locator, options: SyncOptions())274275var doc = try NoteDocument(json: data)          // ProseMirror JSON → lines276doc.toggleMark(.bold, in: 0..<5, at: 2)         // edit277doc.ensureBlockIds()                            // before saving: merges match blocks by id278let body = try doc.jsonData()                   // lines → canonical JSON (sorted keys)279280await engine.schedule()                         // after every local mutation281await engine.sync()                             // app start, foreground, network back, pull-to-refresh282```283284## ⚙️ Configuration285286There is no configuration file. Behaviour is set by types and by a few environment variables used at build or test time.287288| Name | Where | Purpose |289|---|---|---|290| `SPN_ONLY` | `Package.swift` | `Document`, `Store`, `API` or `Sync`: build and test a single module |291| `SPN_LIVE_URL`, `SPN_LIVE_TOKEN` | API and sync tests | enable the live tests against the **test** server; any `*.spboucher.ai` host is refused |292| `SPN_LIVE_TOKEN_DISPOSABLE`, `SPN_LIVE_TOKEN_DISPOSABLE2` | API tests | optional: two more tokens (one is revoked, one logs out) |293| `SPN_RANDOM_SEEDS`, `SPN_RANDOM_ONLY` | sync tests | extra seeds for the randomised convergence test, or a single seed to replay |294| `SPN_DUMP` | document tests | enables the dump test |295296| Type | Settings |297|---|---|298| `APIConfiguration` | `baseURL`, `userAgent`, `requestTimeout`, `resourceTimeout`, `streamIdleTimeout`, `sessionCookieName`, `ceremonyCookieName`; `origin` and `relyingPartyID` are derived from the base URL |299| `SyncOptions` | `autoDownloadLimit` (25 MiB), `includeDatabaseRows` (`false`), `feedPageSize` (200) |300| `KeychainTokenStore` | `service`, `account` (defaults to the origin, so a test server never shares a token with production), `accessGroup`, `useDataProtectionKeychain` |301302## 📝 Module: SPNotesDocument303304Foundation only, value types, all `Sendable` and `Hashable`. Full reference: [`docs/DOCUMENT_API.md`](docs/DOCUMENT_API.md).305306| Area | API (excerpt) |307|---|---|308| Model | `JSONValue`, `PMNode`, `PMMark`, `NoteDocument` (`lines`), `Line` (`displayString`, `inlineLength`) |309| Reading / writing | `NoteDocument(json:)`, `jsonData()` (canonical JSON, sorted keys), `ensureBlockIds()`, `isStructurallyEqual(to:)` |310| Commands | `public mutating func` on `NoteDocument`; they return whether the document changed, never trap on out-of-range input, and leave a document valid for the web schema |311| Marks | `MarkKind`: `bold`, `italic`, `underline`, `strike`, `code`, `highlight`, `subscript`, `superscript`; links through `setLink` / `removeLink` |312| Lists | `ListKind`: `bullet`, `ordered`, `task` |313| Text | `plainText()`, `wordCount`, `preview(maxLength:)`, `pageReferences()`, `attachmentReferences()`, `taskItems()` |314| Export / import | `markdown()`, `html(title:)`, `htmlBody()`, Markdown import |315| Merge | `NoteDocument.merge(base:mine:theirs:)` → `.merged(doc)` or `.conflict` |316| Validation | `schemaViolations(strict:)` |317318<details>319<summary><b>The 30 node types of the web schema known to the core</b></summary>320321`doc`, `text`, `paragraph`, `heading`, `codeBlock`, `footnote`, `detailsSummary`, `bulletList`, `orderedList`,322`taskList`, `listItem`, `taskItem`, `blockquote`, `callout`, `aiBlock`, `details`, `detailsContent`, `table`,323`tableRow`, `tableHeader`, `tableCell`, `image`, `fileBlock`, `blockMath`, `horizontalRule`, `youtube`, `hardBreak`,324`mention`, `inlineMath`, `footnoteRef`.325326Unknown nodes are kept as they are and written back unchanged.327328</details>329330Conventions worth knowing before writing an editor on top of it:331332- **Inline offsets are UTF-16 code units** (`NSString` / `NSRange` compatible). An inline atom (hard break, mention,333  inline math, footnote reference, unknown inline node) counts as exactly **1** unit: display it as one `U+FFFC`334  attachment character.335- New nodes are created with every schema attribute (defaults included) and a fresh block id, exactly as the web editor336  serializes them.337- `plainText`, `wordCount` and `markdown` reproduce the server **byte for byte** (checked on the fixtures and on 400338  random trees against outputs recorded from the web application's own helpers).339- **Three-way merge**: top-level blocks are matched by block id. A block changed, inserted, deleted or moved on one340  side only is taken from that side; the merge never drops a change silently. A top-level block is the whole node (an341  entire list, quote or table): two edits inside the same list conflict.342- HTML export uses the same markup as the server's print view, without KaTeX — math is exported as its LaTeX source.343344## 🗄 Module: SPNotesStore345346The durable local database. Full reference: [`docs/STORE_API.md`](docs/STORE_API.md).347348```swift349public actor NotesStore {350    public init(url: URL, readOnly: Bool = false) throws351    public static func inMemory() throws -> NotesStore352    public func integrityCheck() throws -> [String]     // empty = healthy353    public func checkpoint() throws                     // wal_checkpoint(TRUNCATE)354    public func rebuildSearchIndex() throws355    public func wipeAll() throws                        // "erase this device" only356}357```358359| Aspect | Behaviour |360|---|---|361| Connection | `SQLITE_OPEN_FULLMUTEX`, `journal_mode=WAL`, `synchronous=FULL`, `foreign_keys=ON`, `busy_timeout` 8 s |362| Transactions | every mutation is ONE `BEGIN IMMEDIATE` transaction; an error rolls it back entirely (index, tags, queue and cursor included) and publishes no change event |363| Multi-process | the same file may be opened by the app (read/write), a share extension (read/write) and a widget (`readOnly: true`); a read-only store never creates nor migrates the file |364| Schema | `PRAGMA user_version`, one transaction per migration step; a file newer than the build is refused (`StoreError.schema`) |365| Ids and dates | lowercase UUID strings; timestamps stored as integer milliseconds since the Unix epoch |366| Dirty state | `needsCreate`, `dirtyContent`, `dirtyMeta` (parent, section, position, pinned, archived, icon, cover, inbox), `tagsDirty`, `editSeq` (+1 on every local change to push) |367| Base copy | `PageBase { title, content, properties, version }`: the last synced copy of the version-guarded fields, captured at the first content edit and dropped when the content is clean again |368| Sync state | `conflict` · `error` · `pending` · `synced` |369| Search | FTS5 (`title`, `body`, `tags`), tokenizer `unicode61 remove_diacritics 2`; every term must match, the last one as a prefix; terms are quoted so **no input can cause a syntax error**; bm25 weights title 10, tags 4, body 1; incremental index |370| Ops queue | `trash`, `restore`, `purge`, `deleteAttachment`, `createTask`, `updateTask`, `deleteTask`, `triage`; leased for 120 s; per-target order never broken; backoff 5 s, 10 s, 20 s … capped at 15 min; one idempotency key per op, never changed across retries |371| Change notification | `changes() -> AsyncStream<StoreChange>`: `pages`, `tags`, `attachments`, `queue`, `conflicts`, `reminders`, `meta`, `reset`; emitted after the commit; `noteExternalChange()` when another process wrote the file |372| Errors | `StoreError`: `sqlite`, `notFound`, `invalid`, `cycle`, `readOnly`, `closed`, `schema` |373374## 🌐 Module: SPNotesAPI375376The only module that talks to the backend. Full reference: [`docs/API_CLIENT.md`](docs/API_CLIENT.md); contract:377[`docs/reference/BACKEND_CONTRACT.md`](docs/reference/BACKEND_CONTRACT.md).378379| Type | Role |380|---|---|381| `APIConfiguration` | base URL, `Origin`, User-Agent, timeouts, cookie names |382| `SessionTokenStore`, `KeychainTokenStore`, `InMemoryTokenStore` | where the session token lives |383| `HTTPClient` (actor) | transport: cookies by hand, headers, uploads, downloads, event streams, session events |384| `APIError` | every failure, typed |385| `NotesAPI` | one method per route |386| `PasskeyCeremony`, `PasskeyAuthorizing`, `SystemPasskeyAuthorizer`, `AuthService` | passkeys and sign-in flows |387| `SSEParser`, `AIEvent` | server-sent events of the AI routes |388389Transport rules implemented by `HTTPClient`:390391- One dedicated **ephemeral** `URLSession`: no cookie storage, no URL cache, no credential storage, TLS ≥ 1.2.392  Redirects are refused (the cookie never follows one).393- `Cookie: __Host-note_session=<token>` is written by hand from the token store; a `Set-Cookie` on **any** answer394  (sign-in, step-up rotation) replaces the token and saves it.395- A cleared cookie, or a `401` answering a request that carried the *current* token, clears the store and emits396  `SessionEvent.signedOut` **exactly once** (a late 401 for a token that has since been rotated is ignored).397- The WebAuthn ceremony cookie is captured from the `options` call and sent only on the matching `verify` call, once.398- `Origin` and `Idempotency-Key` only on POST / PATCH / PUT / DELETE; `x-request-id` on every request.399- JSON bodies are encoded with sorted keys (same request → same bytes); answers are decoded with ISO 8601 dates with or400  without fractional seconds, and the PostgreSQL text form that `/api/search` returns.401- User-Agent `SPNotes/<x.y.z> (iPhone|iPad|Macintosh; <os>) native` → the server labels the session402  "SP Notes · iPhone|iPad|Mac".403404Routes covered by `NotesAPI`:405406| Group | Routes |407|---|---|408| Change feed | `GET /api/sync?since=&limit=` |409| Pages | `GET/POST /api/pages`, `GET/PATCH/DELETE/POST /api/pages/<id>`, resolve, versions, tags |410| Attachments | `POST/GET/DELETE /api/attachments`, `GET /api/files/<id>` |411| Authentication | login, reauth (step-up), passkeys, sessions, logout, state |412| AI | chat and inline actions as server-sent events (`data: <json>`, terminator `data: [DONE]`) |413| Other | search, tags, tasks (reminders), trash, inbox and triage, capture, health |414415## 🔄 Module: SPNotesSync416417`SyncEngine` is an actor that keeps one `NotesStore` in step with the backend through `NotesAPI`. Full reference:418[`docs/SYNC_ENGINE.md`](docs/SYNC_ENGINE.md).419420### What the apps call, and when421422| Moment | Call |423|---|---|424| after every local mutation of the store (edit, tag, pin, move, trash, attachment, reminder…) | `await engine.schedule()` (debounced, 2 s by default) and, for the badge, `await engine.refreshCounts()` |425| app start, return to foreground, network back, pull-to-refresh, push notification, after sign-in | `await engine.sync()` |426| after a successful passkey step-up, when `status.needsReauth` was true | `await engine.sync()` (the queued permanent deletions go at once) |427| the user opens a file that is not on the device | `try await engine.download(attachmentId:)` |428| the user settles a conflict | `try await engine.resolveConflict(pageId:choice:)` |429| sign-out, app about to be suspended for good | `await engine.stop()` |430| UI state (spinner, offline banner, counters) | `for await status in await engine.statusUpdates()` |431432`sync()` returns `.completed`, `.offline`, `.signedOut` or `.failed(message)`; nothing is lost in any of these cases.433It is re-entrant: a call made while a cycle runs waits for the **next** cycle, so its outcome always covers the state of434the store at the time of the call.435436### Guarantees4374381. A local edit is acknowledged only for the exact state that was sent (`editSeq`): what is typed while a request is in439   flight stays dirty and goes with the next round or cycle.4402. Local text is replaced in exactly three cases, and each time the replaced text is first saved as a local version:441   an automatic merge, the user choosing "theirs" or "both", a server copy identical to the local one.4423. The server's text is never overwritten without `baseVersion`, and only on top of a copy the local text derives from.4434. A page purged on the server while this device holds unsynced **text** is created again under the same id.4445. Nothing is duplicated by retries (page, file, task, conflict copy).4456. Killing the app at any point loses nothing: all state is in the store.446447### Idempotency448449| Request | What makes a retry safe |450|---|---|451| create page, upload attachment | the client-chosen id (`200` + `x-existing`) — deliberately **no** idempotency key |452| guarded `PATCH` | `baseVersion`; a retry gets `409` with a server copy equal to the local one → adopted |453| unguarded `PATCH`, `PUT tags` | idempotent by nature |454| trash, restore, delete attachment, triage | the op's own key (one key per op, never reused for another payload) |455| create task | `spn-task-<reminder id>` |456| resolve conflict | `spn-res-<page>-<server version>-<choice>-<fingerprint of the local text>` |457458### Conflict policy459460Reconciliation happens when a guarded write is refused (`409`) or when the feed brings a newer version of a note with461unsynced text:4624631. server copy = local copy → adopted, nothing to push;4642. server copy = what this device sent before a lost answer → base moved, local text pushed next;4653. server version lower than the local base (restored server) → the local text is sent again on top, with a snapshot;4664. three-way merge with the stored base copy: title (one side changed), content (top-level blocks by id), properties467   (key by key);4685. otherwise a **conflict is recorded**: the local text is untouched and stays editable, the server copy is stored469   beside it, the note leaves the push list until the user decides.470471| Choice | Note on the server | Where the other text survives |472|---|---|---|473| `.keepMine` | local title and content | server history |474| `.keepTheirs` | unchanged | server history (not applied) and local versions |475| `.keepBoth` | unchanged; the local text becomes a sibling page "… (conflict copy)" | both are notes |476477Flags, tags, moves, icon, cover and position are not versioned: last writer wins, they never conflict.478479<details>480<summary><b>Error classification, trash, purge and server restore</b></summary>481482- **Stops the cycle**: 401 (`.signedOut`); no connectivity, connection failures, timeouts (`.offline`); 429, the third483  5xx of the cycle, a feed that breaks the contract, a closed store (`.failed`); cancellation (`.offline`).484- **Concerns one item**: everything else. A page gets `syncError`; an op gets the store's backoff; an upload refused485  with a 4xx is not sent again before a backoff.486- `403 reauth_required` on a purge is not a failure: the op stays first in line without counting an attempt and487  `needsReauth` stays true.488- Loops are bounded: 4 rounds per cycle; the feed must advance, never go backwards, and may ask for a reset twice at489  most.490- **Trash is not a conflict**: if a note is trashed elsewhere while this device edits it, the edit stays pending;491  restoring the note lets the edit through.492- **Purge on the server**: a clean page disappears; a page with unsynced text is created again with the same id.493- **Purge on the device**: queued; needs a fresh passkey verification.494- **`reset` feed** (the server database was restored): everything is downloaded again; dirty notes are kept and pushed;495  synced notes the restored server no longer has are created again on it. Nothing is deleted on the device in this mode.496497</details>498499## 🗃 Local data model500501Schema version 1 ([`Sources/SPNotesStore/Schema.swift`](Sources/SPNotesStore/Schema.swift)): 10 tables and one FTS5502virtual table.503504| Table | Content |505|---|---|506| `pages` | the notes: server fields, opaque `content` and `properties`, dirty flags, `edit_seq`, sync state, plain text and preview |507| `page_bases` | last synced copy of the version-guarded fields of a page being edited |508| `tags`, `page_tags` | tag names and their pages |509| `pages_fts` | FTS5 index (`title`, `body`, `tags`) |510| `attachments` | file metadata, upload state, local path, last access |511| `ops` | queue of actions that are not dirty state, with their idempotency key and backoff |512| `conflicts` | the server copy stored beside a note in conflict |513| `local_versions` | local history, including every text replaced by a merge or a resolution |514| `reminders` | reminders (pushed to the server as tasks) |515| `meta` | key/value: feed cursor, `sync.sent.*`, `sync.fullCopyPending`… |516517Logical relations (most tables are keyed by the page id rather than by a declared foreign key):518519```mermaid520erDiagram521    pages ||--o| page_bases : "base copy"522    pages ||--o{ page_tags : tagged523    pages ||--o{ attachments : holds524    pages ||--o| conflicts : "server copy"525    pages ||--o{ local_versions : history526    pages ||--o{ reminders : has527    pages ||--|| pages_fts : indexed528```529530Data conventions shared by the native apps: favourite = `properties.favorite === true`; folder = a page with531`properties.folder === true`; reminder = a task `{ pageId, dueAt }`; a PencilKit drawing = an `image` node whose `alt`532is `drawing-<uuid>.png` plus a companion attachment `drawing-<uuid>.pkdrawing` on the same page (editable source).533534## 🔒 Security and privacy535536- The session token lives in the Keychain (`kSecAttrAccessibleAfterFirstUnlockThisDeviceOnly`, not synchronised). It is537  never logged; `APIError` descriptions never contain request data.538- The Keychain account defaults to the origin, so a test server never shares a token with production.539- The `URLSession` is ephemeral, stores no cookie, no cache and no credential, requires TLS ≥ 1.2 and refuses540  redirects.541- Mutating requests carry the exact `Origin` the backend expects; permanent deletions need a passkey verification542  younger than 5 minutes (`reauth_required`).543- Passkeys go through the system (`AuthenticationServices`); `PasskeyCeremony` only does the JSON work.544- Link and image URLs of the HTML export go through the same allow-list as the server; `javascript:` links are dropped545  by the Markdown import.546- The live tests refuse any `*.spboucher.ai` host and clean up what they create.547548## 🧪 Testing549550400 tests (2.0.0, branch `next`): 301 with Swift Testing (`@Test`) and 99 with XCTest (the API module; the 11 live tests are551skipped without `SPN_LIVE_URL`). Per module: Document 131, Store 82, API 99, Sync 67, Design 21. The table below lists the 1.0552suites; 2.0 adds `NextNodeExportTests` (9, one of them over the 4 fixtures published by the backend) and `NextNodeModelTests` (13), `InkTests` (12), `CanvasModelTests` (4),553`CanvasMergeTests` (8), `CanvasLayoutTests` (4) in Document; `MigrationV2Tests` (17), `EntityTests` (15) and the fixture generator554(1, inert without `SPN_MAKE_V1_FIXTURE`) in Store; `RetryTests` (11) in API; `NextNodesSyncTests` (1) and one more queue test in555Sync; `SPNotesDesignTests` (21). What 2.0 changes is described in [`CHANGELOG.md`](CHANGELOG.md), [`docs/MIGRATION-V2.md`](docs/MIGRATION-V2.md)556and [`docs/DESIGN.md`](docs/DESIGN.md).557558| Module | Suite | Tests |559|---|---|---|560| Document (81) | `CommandTests` | 35 |561| | `ModelTests` | 22 |562| | `TextTests` | 16 |563| | `MergeTests` | 7 |564| | `DumpTests` | 1 |565| Store (49) | `SyncTests` | 15 |566| | `PageTests` | 12 |567| | `EngineTests` | 11 |568| | `SearchTests` | 6 |569| | `QueueHistoryTests` | 5 |570| API (88) | `HTTPClientTests` | 22 |571| | `ErrorAndModelTests` | 17 |572| | `PasskeyAndAuthTests` | 15 |573| | `RoutesAndMultipartTests` | 12 |574| | `SSEAndAITests` | 11 |575| | `LiveBackendTests` (only with `SPN_LIVE_URL`) | 11 |576| Sync (65) | `FaultTests` | 26 |577| | `MergeConflictTests` | 16 |578| | `TrashAttachmentQueueTests` | 14 |579| | `SyncEngineTests` | 7 |580| | `RandomConvergenceTests` (seeded) | 1 |581| | `LiveSmokeTests` (only with the live variables) | 1 |582583- **Document**: fixtures recorded from the web application (`Tests/SPNotesDocumentTests/Fixtures`); `swift test` writes584  every document the module emits to `/tmp/spn-docs.json` so they can be validated against the real web schema.585- **API**: a `URLProtocol` stub (`Tests/SPNotesAPITests/TestSupport.swift`).586- **Sync**: an in-memory fake of the backend written from the server's route handlers587  (`Tests/SPNotesSyncTests/FakeBackend.swift`), real store files, 2–3 devices, fault injection, and a seeded randomised588  convergence test.589590Live tests against the test server (never production):591592```sh593ssh -f -N -L 8482:localhost:8482 M4M64c594TOKEN=$(ssh M4M64c 'bash -lc "cd ~/dev/note-native && set -a && . ./.env && set +a && node_modules/.bin/tsx scripts/dev-session.ts"')595SPN_LIVE_URL=http://localhost:8482 SPN_LIVE_TOKEN=$TOKEN SPN_ONLY=API swift test --scratch-path .build-api596```597598A development token is "fresh" for 5 minutes: within that window the purge and revocation paths run for real,599afterwards the tests check the typed `reauth_required` error instead.600601Validating emitted documents against the real web schema (the dev worktree on the backend node, never production):602603```sh604scp /tmp/spn-docs.json M4M64c:/tmp/spn-docs.json605ssh M4M64c 'bash -lc "cd ~/dev/note-v2 && node_modules/.bin/tsx .spn-schema.mts validate /tmp/spn-docs.json"'606```607608Every line must read `ok … identical`.609610## 🧯 Troubleshooting and gotchas611612| Trap | Explanation |613|---|---|614| Never send an idempotency key with a page creation or an upload | The server replays the stored answer of the first attempt for 48 h, whatever the new body, even if the page was purged since. The client-chosen id is what makes the retry safe. |615| The answer of a creation is never trusted blindly | The returned row is always compared with the local one. |616| Call `ensureBlockIds()` before saving | Merges match blocks by id. Notes without ids still work but merge less well. |617| `KeychainTokenStore` fails in an unsigned tool on macOS | The data-protection keychain needs a signed app with a keychain access group; pass `useDataProtectionKeychain: false`. |618| A widget cannot open the store | A read-only store never creates nor migrates the file, and the directory must be writable by the reader (SQLite needs the `-shm` file). |619| The share extension wrote but the app shows nothing | Only changes made through the same `NotesStore` instance are observed: signal the app yourself (Darwin notification) and call `noteExternalChange()`. |620| After `markAttachmentDeleted(id:)` | The app deletes the returned file itself: attachment files belong to the caller. |621| `removeSyncedPages(notIn:)` | Scans every clean page: use it only after a full re-download. |622| `==` on `NoteDocument` | Compares container keys too; use `isStructurallyEqual(to:)` to compare content only. |623| Two parallel module builds | Give each its own `--scratch-path`. |624| Test server | Do not modify its worktree and do not reset its database. |625626<details>627<summary><b>Differences observed between the contract and the test server (2026-11-18)</b></summary>628629- `GET /api/sync` with a cursor ahead of the server answers `{reset: true, cursor: 0, more: true, …}` **without630  `user`**. `limit` bounds each list separately. A non-numeric or negative `since` → `400 bad_cursor`.631- `GET /api/search?mode=quick` returns `updatedAt` in PostgreSQL text form, not ISO 8601.632- Page rows of `POST/GET /api/pages…` carry `workspaceId`, `plainText`, `tsv` and **no `tags`** (the feed has them).633- `POST /api/pages/<id>?action=duplicate` answers `200` (not 201).634- Task `priority` is `low | medium | high` (`400` otherwise).635- `POST /api/inbox/<id>` also returns `created`; `POST /api/inbox/undo` returns `{ok, already}`; triage actions are636  `keep | task | database | reference | archive | trash`.637- `DELETE /api/auth/sessions/<id>` needs a fresh verification (`403 reauth_required`), like the global revocation.638- Idempotency: a replayed answer has `idempotent-replay: true` but **not** `x-existing`; a key reused with a *different639  body* on the same method and path is replayed as well.640- Extra error codes: `bad_request`, `challenge_missing`, `no_instruction` / `bad_language`, `last_passkey` (409),641  `bad_cursor`. `not_enrolled` is a 409. Validation errors are `400` without a code. A malformed UUID in a path answers642  `500`.643- `GET /api/health` is public and does not echo `x-request-id`.644- `GET /api/tags` reports `pages: 0` for tags carried by live pages.645- `DELETE /api/attachments?id=` is a soft delete: `/api/files/<id>` keeps serving the bytes. Purging a page leaves its646  attachments alive with `pageId: null` (no tombstone).647648</details>649650## 🚧 Known limitations651652Documented in the module references.653654**Sync** ([`docs/SYNC_ENGINE.md`](docs/SYNC_ENGINE.md))655656- Reminders are pushed, not pulled: tasks changed or created on the web do not come back.657- Folders purged on the device: the store queues one purge for the root and the server's purge does not cascade, so the658  notes inside stay on the server.659- Moves are last-writer-wins and the server does not check cycles.660- The store has no compare-and-set on its conflict helpers: an edit landing in the few microseconds between the read661  and the write of a merge is replaced by the merge (it is kept as a local version, not lost).662- Undetectable restore: if a restored server has already received enough new writes for its head to pass a device's663  cursor, no `reset` is sent.664- A page that turns into a database row on the web stops being updated on the device (unless `includeDatabaseRows`).665- Pending pages are pushed 500 per round, 4 rounds per cycle; a larger backlog needs another `sync()`.666- Ops refused for good by the server are retried with backoff for ever rather than dropped.667- An edit of the title only, or of the same block, on two devices is a conflict by design.668- `SyncOutcome` has no "cancelled" case: `stop()` reads as `.offline`.669670**Store** ([`docs/STORE_API.md`](docs/STORE_API.md))671672- `trash` / `restore` cancellation ignores pages moved into or out of the subtree between the two calls.673- Properties equality ignores key order but is otherwise structural; content equality is byte equality.674675**Document** ([`docs/DOCUMENT_API.md`](docs/DOCUMENT_API.md))676677- Two documents differing only by Unicode normalization are `==`.678- A lone UTF-16 surrogate escape in JSON is read as U+FFFD.679- Marks are toggled on text only; inline atoms keep their own marks.680- Table helpers address cells by their position in each row; they do not resolve `colspan` / `rowspan` geometry.681- Reference-style links, footnotes and HTML blocks are imported from Markdown as text.682683## 📚 Documentation684685| Document | Content |686|---|---|687| [`docs/DOCUMENT_API.md`](docs/DOCUMENT_API.md) | public API of `SPNotesDocument`: conventions, model, commands, tables, text/Markdown/HTML, merge, validation |688| [`docs/STORE_API.md`](docs/STORE_API.md) | public API and exact semantics of `SPNotesStore` (12 sections) |689| [`docs/API_CLIENT.md`](docs/API_CLIENT.md) | `SPNotesAPI`: set-up, transport rules, errors, endpoints, passkeys, server-sent events, tests |690| [`docs/SYNC_ENGINE.md`](docs/SYNC_ENGINE.md) | the cycle, its guarantees, the conflict policy, what the apps must call, known limits |691| [`docs/reference/BACKEND_CONTRACT.md`](docs/reference/BACKEND_CONTRACT.md) | the backend contract used by the native apps |692| [`docs/reference/README.md`](docs/reference/README.md) | read-only copies of the web sources the core must stay compatible with (`text.ts`, `markdown.ts`, `html.ts`, `validate.ts`, `same-json.ts`, `service.ts`, `web-editor-schema.txt`) |693694## 👤 Author695696<div align="center">697698<img src="docs/brand/icon.png" width="64" height="64" alt="">699700**Simon-Pierre Boucher** — design, development, operations701702[www.spboucher.ai](https://www.spboucher.ai) · [git.spboucher.ai](https://git.spboucher.ai) · contact@spboucher.ai703704</div>705706## 📄 License707708All rights reserved © 2026 Simon-Pierre Boucher.709
Files main
Menu

Theme