Repository Public
1<!--2SP Notes Core3File: docs/SYNC_ENGINE.md4Description: How the synchronisation engine (SPNotesSync) works: the cycle, its guarantees, the conflict policy, what the apps5 must call and when, and the known limits.6Author: Simon-Pierre Boucher7Contact: contact@spboucher.ai8-->910# SPNotesSync — the synchronisation engine1112`SyncEngine` (an actor, `Sources/SPNotesSync/SyncEngine.swift`) keeps one `NotesStore` in step with the backend through13`NotesAPI`. The rule above all others: **no local modification is lost or silently overwritten.** When two devices edit14the same note the changes are merged when that is safe (block by block, title, properties key by key); otherwise both15versions are kept and the user decides.1617```swift18let engine = SyncEngine(store: store, api: NotesAPI(client: client), files: locator, options: SyncOptions())19```2021`AttachmentFileLocating` tells the engine where attachment files live (`url(relativePath:)`,22`relativePath(id:filename:)`, `remove(relativePath:)`); the apps own the directory.2324## 1. What the apps call, and when2526| Moment | Call |27|---|---|28| after every local mutation of the store (edit, tag, pin, move, trash, attachment, reminder…) | `await engine.schedule()` (debounced, 2 s by default) and, for the badge, `await engine.refreshCounts()` |29| app start, return to foreground, network back, pull-to-refresh, push notification, after sign-in | `await engine.sync()` |30| after a successful passkey step-up, when `status.needsReauth` was true | `await engine.sync()` (the queued permanent deletions go at once) |31| the user opens a file that is not on the device | `try await engine.download(attachmentId:)` |32| the user settles a conflict | `try await engine.resolveConflict(pageId:choice:)` |33| sign-out, app about to be suspended for good | `await engine.stop()` |34| UI state (spinner, offline banner, counters) | `for await status in await engine.statusUpdates()` |3536- `sync()` returns `.completed`, `.offline` (no network, unreachable server, timeout, or `stop()`), `.signedOut` (no37 token, or a 401 during the cycle — the HTTP client has then forgotten the token) or `.failed(message)` (server errors,38 a server that breaks the contract, a closed store). Nothing is lost in any of these cases: what was not acknowledged39 stays pending and the next cycle resumes.40- `sync()` is re-entrant: a call made while a cycle runs waits for the **next** cycle, so its outcome always covers the41 state of the store at the time of the call; any number of such calls share that single extra cycle. If the running42 cycle does not complete, all the waiting callers get its outcome and no doomed cycle is started.43- Before editing documents, call `NoteDocument.ensureBlockIds()` and save `jsonData()`: merges match blocks by id.44 Notes without ids still work (the ids the server assigns are adopted after the creation), but they merge less well.45- Queue an inbox triage with `store.setInbox(id:at: nil)` + `store.enqueueOp(kind: OpKind.triage, targetId: id,46 payload: JSONEncoder().encode(QueuedTriage(action: "keep" | "archive" | "trash", section:, parentId:)))`.47- After `store.markAttachmentDeleted(id:)` the app deletes the returned file itself.4849`SyncStatus`: `phase` (`idle | syncing | offline | signedOut | failed`), `lastSuccess`, `pending` (dirty notes — notes50in conflict included —, queued actions and files waiting for upload, files whose local file is missing excepted),51`conflicts`, `needsReauth`.5253## 2. The cycle5455One cycle = up to 4 rounds of **push then pull**, then background downloads.5657**Push**58591. *Creations*, parents before children (`POST /api/pages` with the local id, then one unguarded `PATCH` for what a60 creation cannot carry — pin, archive, cover, position — and `PUT tags`). A page whose parent is still to be created61 waits: the server would file it at the top level.622. *Uploads* (`POST /api/attachments` with the local id). A file waits until its page exists on the server.633. *Edits of known pages*: unguarded fields (`dirtyMeta`) and tags first, then title/content/properties with64 `baseVersion`. `409` → reconciliation (§4). `404` → `GET ?head=1` tells trash from purge (§5).654. *Queued actions* (trash, restore, purge, attachment deletion, triage) in order, each with the idempotency key stored66 with the op, then *reminders* (tasks).6768**Pull** — `GET /api/sync` from the stored cursor until `more` is false; each answer is applied in one store69transaction with its cursor. Divergences reported by the store are reconciled (§4). Files orphaned by tombstones, and70local files of attachments deleted on the server, are removed.7172A further round runs when the previous one left something to push (a merge, a full batch of 500 pages, a restore that73unblocked a note). **Downloads** then fetch up to 40 wanted files (pictures and `.pkdrawing` sources up to74`autoDownloadLimit`, and files marked keep-offline) for notes that are not in the trash.7576### Idempotency7778| Request | What makes a retry safe |79|---|---|80| create page, upload attachment | the client-chosen id (`200` + `x-existing`). **No idempotency key**: the server replays the stored answer of the first attempt for 48 h, whatever the new body, even if the page was purged since. |81| guarded `PATCH` | `baseVersion`; a retry gets `409` with a server copy equal to the local one → adopted |82| unguarded `PATCH`, `PUT tags` | idempotent by nature |83| trash, restore, delete attachment, triage | the op's own key (one key per op, never reused for another payload) |84| create task | `spn-task-<reminder id>`; the returned task is compared with the reminder and patched if it differs (a replay describes the first attempt) |85| resolve conflict | `spn-res-<page>-<server version>-<choice>-<fingerprint of the local text>` |8687The answer of a creation is **never trusted blindly**: the returned row is always compared with the local one.8889### Lost acknowledgements9091Before a guarded write the engine stores (store meta `sync.sent.<page id>`) the version the text will have on the92server and its fingerprint. If the answer is lost and the user keeps typing — even in the same paragraph, even after93the app was killed — the server copy that carries that fingerprint at that version is recognised as this device's own94write: it becomes the base, the later typing goes on top, and no conflict is raised.9596### Error classification9798- **Stops the cycle**: 401 (`.signedOut`); no connectivity, connection failures, timeouts (`.offline`); 429, the third99 5xx of the cycle, a feed that breaks the contract, a closed store (`.failed`); cancellation (`.offline`).100- **Concerns one item**: everything else. A page gets `syncError` (state `error`, retried at the next cycle); an op gets101 the store's backoff (5 s … 15 min); an upload refused with a 4xx is not sent again before a backoff; reminders are102 retried at the next cycle.103- `403 reauth_required` on a purge: not a failure. The op stays first in line without counting an attempt,104 `needsReauth` stays true from cycle to cycle, the rest of the queue goes on.105- Loops are bounded: 4 rounds per cycle; the feed must advance (`cursor` strictly greater while `more`), never go106 backwards, and may ask for a reset twice at most; otherwise the cycle fails.107108## 3. Guarantees1091101. A local edit is acknowledged only for the exact state that was sent (`editSeq`): what is typed while a request is in111 flight stays dirty and goes with the next round or cycle.1122. Local text is replaced in exactly three cases, and each time the replaced text is first saved as a local version113 (`conflict-local`) by the store: an automatic merge, the user choosing "theirs" or "both", a server copy identical114 to the local one. (After a creation the device also adopts the block ids the server assigned: same text.)1153. The server's text is never overwritten without `baseVersion`, and only on top of a copy the local text derives from:116 the acknowledged base, a merge with the current copy, or this device's own earlier write. The one deliberate117 exception is a restored server (§5): the local text is sent on top of the older copy and the server is asked to118 snapshot what is replaced (`versionReason`).1194. A page purged on the server while this device holds unsynced **text** is created again under the same id.1205. Nothing is duplicated by retries (page, file, task, conflict copy).1216. Killing the app at any point loses nothing: all state is in the store (dirty flags, bases, queue, cursor,122 `sync.sent.*`, `sync.fullCopyPending`).123124Covered by `Tests/SPNotesSyncTests` (in-memory fake of the backend written from the server's route handlers, real125store files, 2–3 devices, fault injection, a seeded randomised convergence test) and by a live smoke test against the126test server.127128## 4. Conflict policy129130Reconciliation happens when a guarded write is refused (409) or when the feed brings a newer version of a note with131unsynced text. It always works on the local row **as it is at that moment** (re-read after the network call):1321331. server copy = local copy → adopted, nothing to push;1342. server copy = what this device sent before a lost answer → base moved, local text pushed next;1353. server version lower than the local base (restored server) → the local text is sent again on top, with a snapshot;1364. three-way merge with the stored base copy: title (one side changed), content (`NoteDocument.merge`, top-level blocks137 by id), properties (key by key). Success → the merge replaces the local text and is pushed in the next round;1385. otherwise a **conflict is recorded**: the local text is untouched and stays editable, the server copy is stored139 beside it (`store.conflict(pageId:)`), the note leaves the push list until the user decides. A newer server copy140 replaces the record; a server copy that becomes identical to the local text dissolves it.141142A page that has no base (never acknowledged) and meets a different text under its id is a conflict too.143144`resolveConflict(pageId:choice:)` (online):145146| Choice | Note on the server | Where the other text survives |147|---|---|---|148| `.keepMine` | local title and content | server history ("conflict · other session's copy") |149| `.keepTheirs` | unchanged | local text: server history (not applied) and local versions |150| `.keepBoth` | unchanged; the local text becomes a sibling page "… (conflict copy)" (its id is returned) | both are notes; the local text is also in the local versions |151152- `.keepMine` keeps the local **properties** (the server's resolve route only takes title and content) and whatever153 was typed during the request: both go with the next push.154- `409` from the server (the note moved again while the user was deciding): the newer copy is recorded and the error is155 thrown — show the new copy and ask again. If that copy equals the local text the conflict is simply closed.156- Flags, tags, moves, icon, cover and position are not versioned: last writer wins, they never conflict.157- While a note is in conflict none of its changes are pushed (flags and tags included).158159## 5. Trash, purge, server restore160161- **Trash is not a conflict.** The server refuses every write on a trashed page. If a note is trashed elsewhere while162 this device edits it, the edit stays pending (no error); the feed puts the note in the trash here too. Restoring it163 (from any device) lets the edit through — from this device, in the same `sync()` call. If, after the feed was read,164 the note is still *live* on this device (it was restored here together with a folder the server does not know it165 belongs to yet), the engine restores it on the server: what the user sees wins.166- **Purge on the server**: a clean page disappears; a page with unsynced text is created again with the same id; a page167 with only unsynced flags, tags or a move follows the purge.168- **Purge on the device**: queued; needs a fresh passkey verification (`needsReauth`). A page purged locally while its169 creation was in flight is trashed on the server at once and its purge is queued.170- An action on a page that is not created on the server yet waits for the creation.171- **`reset` feed** (the server database was restored): everything is downloaded again (an interrupted copy starts over);172 dirty notes are kept and pushed; synced notes the restored server no longer has are **created again** on it; notes173 it has in an older version are sent again. Nothing is deleted on the device in this mode.174- **Full copy from cursor 0 without reset** (first sync, or the app set the cursor to 0): synced pages absent from the175 complete copy are removed (purged while the device was away).176177## 6. Known limits178179- **Reminders are pushed, not pulled**: tasks changed or created on the web do not come back (the feed has no tasks and180 the store has no "apply remote reminder").181- **Folders purged on the device**: the store queues one purge for the root; the server's purge does not cascade, so182 the notes inside stay on the server (orphans) although they are gone from the device. Needs a fix in the store (one183 op per page) or in the backend.184- **Moves are last-writer-wins and the server does not check cycles**: two devices moving two folders into each other185 offline produce a cycle on the server.186- **Compare-and-set**: the store has no "apply only if `editSeq` is still N" on its conflict helpers. The engine187 re-reads the row right before applying a merge, but an edit landing in the few microseconds between that read and the188 write is replaced by the merge (it is kept as a local version, not lost).189- **Undetectable restore**: if a restored server has already received enough new writes for its head to pass a190 device's cursor, no `reset` is sent and the device cannot know. Notes created after the backup stay on the device191 only (and a later forced full copy from cursor 0 would remove them).192- A page that turns into a database row on the web stops being updated on the device (rows are filtered out unless193 `includeDatabaseRows`).194- Pending pages are pushed 500 per round, 4 rounds per cycle; a larger backlog needs another `sync()` (`pending` says so).195- Ops refused for good by the server (4xx other than 404/409 `not_in_inbox`) are retried with backoff for ever rather196 than dropped: a user action is never discarded silently.197- An edit of the title only, or of the same block, on two devices is a conflict by design, even for a one-letter change.198- `SyncOutcome` has no "cancelled" case: `stop()` reads as `.offline`.199- **2.0 — unknown queued actions are kept.** Until 2.0 an op whose kind the engine did not know (written by a newer build, or200 the never-dispatched `createTask` / `updateTask` / `deleteTask`) was completed, i.e. silently deleted. It is now parked201 (`NotesStore.parkOp`): it stays in the queue with its payload and idempotency key, is reported in `SyncStatus.unsupported`202 (and no longer in `pending`), does not hold back later actions on the same target, and goes back in line at the start of a203 cycle of a build whose `SyncEngine.supportedOpKinds` contains its kind.204- **Reminders stay push-only (decision of 2.0, see §7.6).**205206## 7. Entity sync (2.0 — API-NEXT §3, §4.2, §4.8)207208Everything below runs only against a server that advertises the features in `GET /api/meta`. Against a 2.x server (404 on209`/api/meta`) the engine sends exactly the requests of 1.0, plus that one `GET /api/meta` per `SyncOptions.metaMaxAge` (10 min):210no `kinds`, no 3.0 route; queued task operations and edited canvases simply wait (they count in `status.pending`) until a server211that knows them answers. The whole test suite runs green in that mode (`SPN_FAKE_API_LEVEL=2`).212213### 7.1 Features214215`engine.serverFeatures()` → `ServerMeta?` (nil before the first cycle, `.legacy` for 2.x). The answer is cached in the store216(`CacheKey.meta`) so that the repositories know the features offline. Flags used by the engine: `entities` (feed with kinds),217`tasks-v2` (task outbox), `canvas` (canvas push).218219### 7.2 The feed with kinds, and the cursor rule220221With `entities`, every `GET /api/sync` carries `kinds=` (`SyncOptions.entityKinds`, default all nine: `task, database,222database_property, database_view, database_row, edge, reference, canvas, concept`, in the server's order).223224**A cursor is only valid for the set of kinds it was obtained with** (API-NEXT §3). The engine stores the set next to the cursor225(store meta `sync.kinds`); whenever the set to ask differs from the stored one — first run of 2.0 against a 3.0 server, a server226that gains or loses the feature, an app that changes `entityKinds` — the cursor goes back to 0 and the feed is read again in full227(pages already known are overwritten with identical data; the usual "complete copy from 0" rules apply). There is one cursor;228`sync_cursors` mirrors it per kind for inspection.229230Applying one answer:2312321. entity rows → `NotesStore.applyRemoteEntities` (one transaction; a row whose stored `rev` is not lower is skipped, so a window233 applied twice changes nothing — the entities are written before the pages, whose transaction carries the cursor);2342. a row with `deletedAt` (soft delete: tasks, databases, database rows, references) leaves the local copy like a tombstone;2353. tombstones of any kind other than `page` / `attachment` are applied to `entities` (before 2.0 they were skipped);2364. `canvas` rows go to the `canvases` table (`applyRemoteCanvas`), where a local edit meets the server copy (§7.4); a `canvas`237 tombstone removes a clean canvas (its page tombstone comes in the same answer);2385. a row that has local changes pending keeps its local JSON (the server copy is kept aside in `remoteJSON`).239240`reset` (cursor ahead of the server): entity revisions are zeroed too (`resetEntitySync`), everything is read again, nothing is241deleted locally, queued operations are kept and sent. After a complete copy from 0 **without** reset, clean entities absent from242the copy are removed (deleted while the device missed the tombstones); rows with local work are kept.243244### 7.3 Tasks: the outbox245246Tasks are written offline-first by `TasksRepository` (docs/REPOSITORIES.md): the local row and an `entity_ops` entry in one store247transaction, under a client-chosen UUID. `pushEntityOps` sends them in outbox order:248249| Verb | Request | What makes a retry safe |250|---|---|---|251| `create` | `POST /api/tasks {id, …}` | the client id (`200` + `x-existing`: the stored task, untouched). No idempotency key, for the same reason as pages. |252| `update` | `PATCH /api/tasks/<id>` with only the changed fields | the op's own idempotency key: an update whose answer was lost is applied once, never again on top of a later change |253| `delete` | `DELETE /api/tasks/<id>` | idempotent on the server |254255- **Order.** The operations of one task are never reordered: when one does not go through, the later ones of that task wait; other256 tasks go on. A task anchored to a page that is still to be created waits for the page (same `sync()` call).257- **Two devices.** Updates carry fields, not rows: different fields of the same task both win; the same field goes to the last258 request to reach the server. A deletion wins over an edit made elsewhere (`404` on the update → nothing left to do).259- **Settling.** While operations of a task wait, its local row stays as the user left it. When the outbox of the task is empty the260 row takes the server's copy: the answer just received, or — when the feed brought a newer revision meanwhile, or the answer may261 be a replay — a fresh `GET /api/tasks/<id>`. A task acknowledged but not settled (app killed between the two steps, the fresh262 read failed) is settled at the next cycle.263- **Errors.** Offline, timeouts, 401, 429, the third 5xx: the cycle stops, nothing is lost. A 5xx on one operation: backoff264 (5 s … 15 min, `expediteEntityOps` ends it). `400 bad_page` (the page is in the server's trash): backoff, not refused — it goes265 when the page is back. Any other 4xx / 409: the operation is **rejected** — kept with its payload and its key,266 counted in `SyncStatus.rejected`, never retried by itself, never dropped; `TasksRepository.retry / discard` decide.267- An operation of a kind or verb this build does not execute is marked `unsupported` (kept, `SyncStatus.unsupported`).268 `SyncEngine.supportedEntityOps` lists what runs.269270### 7.4 Canvases271272A canvas is a page of type `canvas` plus its `spn-canvas-1` document in the `canvases` table. `pushCanvases`:2732741. the page first (a 3.0 server creates the empty canvas, version 1, with the page);2752. a canvas never synced (`version == 0`): `GET /api/canvases/<id>`, merged with the local drawing, sent by the next round;2763. `PUT /api/canvases/<id> {data, baseVersion}` → `200` → `markCanvasPushed` (what was drawn during the request stays dirty);2774. `409 conflict` (or a feed row for a canvas with local changes) → **three-way merge by node / edge id**278 (`CanvasDocument.merge`, base = the last copy common with the server):279 - the server already holds the local canvas (a save whose answer was lost) → adopted, no conflict, no extra version;280 - clean merge → it replaces the local canvas and is pushed on top of the server version, in the same `sync()` call;281 - merge with conflicts (same card changed differently; edited on one side and deleted on the other — SPEC §7: the edited card282 is kept) → **nothing is changed on either side**: the server copy is stored beside the local canvas283 (`recordCanvasConflict`), which stays editable and leaves the push list; `SyncStatus.canvasConflicts` counts it;284 `CanvasRepository.conflict / resolve` let the user decide (mine, theirs, or a merged document). A newer server copy that285 merges cleanly dissolves the conflict.2865. `404` (the page is in the server's trash) / `400`, `413 bad_canvas`: the canvas keeps its changes, `syncError` says why, and287 it is tried again at the next cycle.288289Every write of a merge is a compare-and-set on the canvas `editSeq`: a merge computed on a copy the user has since changed is290computed again, never written over the newer drawing.291292### 7.5 Guarantees added by 2.02932947. A task created offline exists exactly once on the server, under its id, whatever is retried.2958. No task operation is dropped: it is acknowledged, waiting, rejected (visible) or unsupported (visible).2969. A canvas is never overwritten without `baseVersion`; neither the local nor the server copy is replaced by a merge that has297 conflicts; a card nobody removed never disappears.298299Covered by `EntityFeedTests` (8), `TaskSyncTests` (12), `CanvasSyncTests` (9), `EntityConvergenceTests` (seeded: tasks and300canvases on two devices plus the web, lost answers, 5xx, network loss, restarts mid-queue) and `LiveEntitySyncTests` against the301scratch backend.302303### 7.6 Reminders304305The 1.0 reminder path (`store.dirtyReminders()` → `POST/PATCH/DELETE /api/tasks`, server-assigned ids, push only) is unchanged.306Turning it into task sync was examined and **not done, because the mapping is not safe**: (1) existing reminders are linked to307their task by a server-assigned id (`serverTaskId`), not by a client id, and the store has no "apply remote reminder"; (2) a308reminder drives a local notification — pulling every dated task of the workspace into reminders would schedule notifications309nobody asked for on that device; (3) a reminder has no status `doing`, priority or project, so a round trip would lose task310fields. What 2.0 gives instead: the tasks created by reminders come back through the feed like any task (`TasksRepository`311shows them, with their page anchor), and an app that wants a reminder for a task creates it explicitly. A new app feature should312use `TasksRepository` for tasks and keep reminders for local alarms.313314### 7.7 Limits of 2.0315316- Databases, properties, views, rows, relations, concepts and references are **read offline, written online** (SPEC §6): the317 repositories call the server and update the local copy from the answer; offline they throw `RepositoryError.offline`. Nothing318 of those kinds is ever queued by this build.319- Task conflicts are field-level last-writer-wins; there is no "both" copy of a task.320- The answer of `GET /api/sync` with `reset: true` carries empty lists per kind (the contract says `"entities": {}`): harmless.321- The fake's `purge` keeps the 2.x cascade rules for pages (the 3.0 server also purges sub-pages trashed with the page and their322 attachments, API-NEXT §4.15); the engine relies on the tombstones either way.323- Canvas page conflicts on the *page row* (title) follow the page rules; only `data` is merged by id.324