SPNotesSync — the synchronisation engine
SyncEngine (an actor, Sources/SPNotesSync/SyncEngine.swift) keeps one NotesStore in step with the backend through
NotesAPI. The rule above all others: no local modification is lost or silently overwritten. When two devices edit
the same note the changes are merged when that is safe (block by block, title, properties key by key); otherwise both
versions are kept and the user decides.
let engine = SyncEngine(store: store, api: NotesAPI(client: client), files: locator, options: SyncOptions())AttachmentFileLocating tells the engine where attachment files live (url(relativePath:),
relativePath(id:filename:), remove(relativePath:)); the apps own the directory.
1. What the apps call, and when
| Moment | Call |
|---|---|
| after every local mutation of the store (edit, tag, pin, move, trash, attachment, reminder…) | await engine.schedule() (debounced, 2 s by default) and, for the badge, await engine.refreshCounts() |
| app start, return to foreground, network back, pull-to-refresh, push notification, after sign-in | await engine.sync() |
after a successful passkey step-up, when status.needsReauth was true |
await engine.sync() (the queued permanent deletions go at once) |
| the user opens a file that is not on the device | try await engine.download(attachmentId:) |
| the user settles a conflict | try await engine.resolveConflict(pageId:choice:) |
| sign-out, app about to be suspended for good | await engine.stop() |
| UI state (spinner, offline banner, counters) | for await status in await engine.statusUpdates() |
sync()returns.completed,.offline(no network, unreachable server, timeout, orstop()),.signedOut(no token, or a 401 during the cycle — the HTTP client has then forgotten the token) or.failed(message)(server errors, a server that breaks the contract, a closed store). Nothing is lost in any of these cases: what was not acknowledged stays pending and the next cycle resumes.sync()is re-entrant: a call made while a cycle runs waits for the next cycle, so its outcome always covers the state of the store at the time of the call; any number of such calls share that single extra cycle. If the running cycle does not complete, all the waiting callers get its outcome and no doomed cycle is started.- Before editing documents, call
NoteDocument.ensureBlockIds()and savejsonData(): merges match blocks by id. Notes without ids still work (the ids the server assigns are adopted after the creation), but they merge less well. - Queue an inbox triage with
store.setInbox(id:at: nil)+store.enqueueOp(kind: OpKind.triage, targetId: id, payload: JSONEncoder().encode(QueuedTriage(action: "keep" | "archive" | "trash", section:, parentId:))). - After
store.markAttachmentDeleted(id:)the app deletes the returned file itself.
SyncStatus: phase (idle | syncing | offline | signedOut | failed), lastSuccess, pending (dirty notes — notes
in conflict included —, queued actions and files waiting for upload, files whose local file is missing excepted),
conflicts, needsReauth.
2. The cycle
One cycle = up to 4 rounds of push then pull, then background downloads.
Push
- Creations, parents before children (
POST /api/pageswith the local id, then one unguardedPATCHfor what a creation cannot carry — pin, archive, cover, position — andPUT tags). A page whose parent is still to be created waits: the server would file it at the top level. - Uploads (
POST /api/attachmentswith the local id). A file waits until its page exists on the server. - Edits of known pages: unguarded fields (
dirtyMeta) and tags first, then title/content/properties withbaseVersion.409→ reconciliation (§4).404→GET ?head=1tells trash from purge (§5). - Queued actions (trash, restore, purge, attachment deletion, triage) in order, each with the idempotency key stored with the op, then reminders (tasks).
Pull — GET /api/sync from the stored cursor until more is false; each answer is applied in one store
transaction with its cursor. Divergences reported by the store are reconciled (§4). Files orphaned by tombstones, and
local files of attachments deleted on the server, are removed.
A further round runs when the previous one left something to push (a merge, a full batch of 500 pages, a restore that
unblocked a note). Downloads then fetch up to 40 wanted files (pictures and .pkdrawing sources up to
autoDownloadLimit, and files marked keep-offline) for notes that are not in the trash.
Idempotency
| Request | What makes a retry safe |
|---|---|
| create page, upload attachment | the client-chosen id (200 + x-existing). No idempotency key: the server replays the stored answer of the first attempt for 48 h, whatever the new body, even if the page was purged since. |
guarded PATCH |
baseVersion; a retry gets 409 with a server copy equal to the local one → adopted |
unguarded PATCH, PUT tags |
idempotent by nature |
| trash, restore, delete attachment, triage | the op's own key (one key per op, never reused for another payload) |
| create task | spn-task-<reminder id>; the returned task is compared with the reminder and patched if it differs (a replay describes the first attempt) |
| resolve conflict | spn-res-<page>-<server version>-<choice>-<fingerprint of the local text> |
The answer of a creation is never trusted blindly: the returned row is always compared with the local one.
Lost acknowledgements
Before a guarded write the engine stores (store meta sync.sent.<page id>) the version the text will have on the
server and its fingerprint. If the answer is lost and the user keeps typing — even in the same paragraph, even after
the app was killed — the server copy that carries that fingerprint at that version is recognised as this device's own
write: it becomes the base, the later typing goes on top, and no conflict is raised.
Error classification
- Stops the cycle: 401 (
.signedOut); no connectivity, connection failures, timeouts (.offline); 429, the third 5xx of the cycle, a feed that breaks the contract, a closed store (.failed); cancellation (.offline). - Concerns one item: everything else. A page gets
syncError(stateerror, retried at the next cycle); an op gets the store's backoff (5 s … 15 min); an upload refused with a 4xx is not sent again before a backoff; reminders are retried at the next cycle. 403 reauth_requiredon a purge: not a failure. The op stays first in line without counting an attempt,needsReauthstays true from cycle to cycle, the rest of the queue goes on.- Loops are bounded: 4 rounds per cycle; the feed must advance (
cursorstrictly greater whilemore), never go backwards, and may ask for a reset twice at most; otherwise the cycle fails.
3. Guarantees
- A local edit is acknowledged only for the exact state that was sent (
editSeq): what is typed while a request is in flight stays dirty and goes with the next round or cycle. - Local text is replaced in exactly three cases, and each time the replaced text is first saved as a local version
(
conflict-local) by the store: an automatic merge, the user choosing "theirs" or "both", a server copy identical to the local one. (After a creation the device also adopts the block ids the server assigned: same text.) - The server's text is never overwritten without
baseVersion, and only on top of a copy the local text derives from: the acknowledged base, a merge with the current copy, or this device's own earlier write. The one deliberate exception is a restored server (§5): the local text is sent on top of the older copy and the server is asked to snapshot what is replaced (versionReason). - A page purged on the server while this device holds unsynced text is created again under the same id.
- Nothing is duplicated by retries (page, file, task, conflict copy).
- Killing the app at any point loses nothing: all state is in the store (dirty flags, bases, queue, cursor,
sync.sent.*,sync.fullCopyPending).
Covered by Tests/SPNotesSyncTests (in-memory fake of the backend written from the server's route handlers, real
store files, 2–3 devices, fault injection, a seeded randomised convergence test) and by a live smoke test against the
test server.
4. Conflict policy
Reconciliation happens when a guarded write is refused (409) or when the feed brings a newer version of a note with unsynced text. It always works on the local row as it is at that moment (re-read after the network call):
- server copy = local copy → adopted, nothing to push;
- server copy = what this device sent before a lost answer → base moved, local text pushed next;
- server version lower than the local base (restored server) → the local text is sent again on top, with a snapshot;
- three-way merge with the stored base copy: title (one side changed), content (
NoteDocument.merge, top-level blocks by id), properties (key by key). Success → the merge replaces the local text and is pushed in the next round; - otherwise a conflict is recorded: the local text is untouched and stays editable, the server copy is stored
beside it (
store.conflict(pageId:)), the note leaves the push list until the user decides. A newer server copy replaces the record; a server copy that becomes identical to the local text dissolves it.
A page that has no base (never acknowledged) and meets a different text under its id is a conflict too.
resolveConflict(pageId:choice:) (online):
| Choice | Note on the server | Where the other text survives |
|---|---|---|
.keepMine |
local title and content | server history ("conflict · other session's copy") |
.keepTheirs |
unchanged | local text: server history (not applied) and local versions |
.keepBoth |
unchanged; the local text becomes a sibling page "… (conflict copy)" (its id is returned) | both are notes; the local text is also in the local versions |
.keepMinekeeps the local properties (the server's resolve route only takes title and content) and whatever was typed during the request: both go with the next push.409from the server (the note moved again while the user was deciding): the newer copy is recorded and the error is thrown — show the new copy and ask again. If that copy equals the local text the conflict is simply closed.- Flags, tags, moves, icon, cover and position are not versioned: last writer wins, they never conflict.
- While a note is in conflict none of its changes are pushed (flags and tags included).
5. Trash, purge, server restore
- Trash is not a conflict. The server refuses every write on a trashed page. If a note is trashed elsewhere while
this device edits it, the edit stays pending (no error); the feed puts the note in the trash here too. Restoring it
(from any device) lets the edit through — from this device, in the same
sync()call. If, after the feed was read, the note is still live on this device (it was restored here together with a folder the server does not know it belongs to yet), the engine restores it on the server: what the user sees wins. - Purge on the server: a clean page disappears; a page with unsynced text is created again with the same id; a page with only unsynced flags, tags or a move follows the purge.
- Purge on the device: queued; needs a fresh passkey verification (
needsReauth). A page purged locally while its creation was in flight is trashed on the server at once and its purge is queued. - An action on a page that is not created on the server yet waits for the creation.
resetfeed (the server database was restored): everything is downloaded again (an interrupted copy starts over); dirty notes are kept and pushed; synced notes the restored server no longer has are created again on it; notes it has in an older version are sent again. Nothing is deleted on the device in this mode.- Full copy from cursor 0 without reset (first sync, or the app set the cursor to 0): synced pages absent from the complete copy are removed (purged while the device was away).
6. Known limits
- Reminders are pushed, not pulled: tasks changed or created on the web do not come back (the feed has no tasks and the store has no "apply remote reminder").
- Folders purged on the device: the store queues one purge for the root; the server's purge does not cascade, so the notes inside stay on the server (orphans) although they are gone from the device. Needs a fix in the store (one op per page) or in the backend.
- Moves are last-writer-wins and the server does not check cycles: two devices moving two folders into each other offline produce a cycle on the server.
- Compare-and-set: the store has no "apply only if
editSeqis still N" on its conflict helpers. The engine re-reads the row right before applying a merge, but an edit landing in the few microseconds between that read and the write is replaced by the merge (it is kept as a local version, not lost). - Undetectable restore: if a restored server has already received enough new writes for its head to pass a
device's cursor, no
resetis sent and the device cannot know. Notes created after the backup stay on the device only (and a later forced full copy from cursor 0 would remove them). - A page that turns into a database row on the web stops being updated on the device (rows are filtered out unless
includeDatabaseRows). - Pending pages are pushed 500 per round, 4 rounds per cycle; a larger backlog needs another
sync()(pendingsays so). - Ops refused for good by the server (4xx other than 404/409
not_in_inbox) are retried with backoff for ever rather than dropped: a user action is never discarded silently. - An edit of the title only, or of the same block, on two devices is a conflict by design, even for a one-letter change.
SyncOutcomehas no "cancelled" case:stop()reads as.offline.- 2.0 — unknown queued actions are kept. Until 2.0 an op whose kind the engine did not know (written by a newer build, or
the never-dispatched
createTask/updateTask/deleteTask) was completed, i.e. silently deleted. It is now parked (NotesStore.parkOp): it stays in the queue with its payload and idempotency key, is reported inSyncStatus.unsupported(and no longer inpending), does not hold back later actions on the same target, and goes back in line at the start of a cycle of a build whoseSyncEngine.supportedOpKindscontains its kind. - Reminders stay push-only (decision of 2.0, see §7.6).
7. Entity sync (2.0 — API-NEXT §3, §4.2, §4.8)
Everything below runs only against a server that advertises the features in GET /api/meta. Against a 2.x server (404 on
/api/meta) the engine sends exactly the requests of 1.0, plus that one GET /api/meta per SyncOptions.metaMaxAge (10 min):
no kinds, no 3.0 route; queued task operations and edited canvases simply wait (they count in status.pending) until a server
that knows them answers. The whole test suite runs green in that mode (SPN_FAKE_API_LEVEL=2).
7.1 Features
engine.serverFeatures() → ServerMeta? (nil before the first cycle, .legacy for 2.x). The answer is cached in the store
(CacheKey.meta) so that the repositories know the features offline. Flags used by the engine: entities (feed with kinds),
tasks-v2 (task outbox), canvas (canvas push).
7.2 The feed with kinds, and the cursor rule
With entities, every GET /api/sync carries kinds= (SyncOptions.entityKinds, default all nine: task, database, database_property, database_view, database_row, edge, reference, canvas, concept, in the server's order).
A cursor is only valid for the set of kinds it was obtained with (API-NEXT §3). The engine stores the set next to the cursor
(store meta sync.kinds); whenever the set to ask differs from the stored one — first run of 2.0 against a 3.0 server, a server
that gains or loses the feature, an app that changes entityKinds — the cursor goes back to 0 and the feed is read again in full
(pages already known are overwritten with identical data; the usual "complete copy from 0" rules apply). There is one cursor;
sync_cursors mirrors it per kind for inspection.
Applying one answer:
- entity rows →
NotesStore.applyRemoteEntities(one transaction; a row whose storedrevis not lower is skipped, so a window applied twice changes nothing — the entities are written before the pages, whose transaction carries the cursor); - a row with
deletedAt(soft delete: tasks, databases, database rows, references) leaves the local copy like a tombstone; - tombstones of any kind other than
page/attachmentare applied toentities(before 2.0 they were skipped); canvasrows go to thecanvasestable (applyRemoteCanvas), where a local edit meets the server copy (§7.4); acanvastombstone removes a clean canvas (its page tombstone comes in the same answer);- a row that has local changes pending keeps its local JSON (the server copy is kept aside in
remoteJSON).
reset (cursor ahead of the server): entity revisions are zeroed too (resetEntitySync), everything is read again, nothing is
deleted locally, queued operations are kept and sent. After a complete copy from 0 without reset, clean entities absent from
the copy are removed (deleted while the device missed the tombstones); rows with local work are kept.
7.3 Tasks: the outbox
Tasks are written offline-first by TasksRepository (docs/REPOSITORIES.md): the local row and an entity_ops entry in one store
transaction, under a client-chosen UUID. pushEntityOps sends them in outbox order:
| Verb | Request | What makes a retry safe |
|---|---|---|
create |
POST /api/tasks {id, …} |
the client id (200 + x-existing: the stored task, untouched). No idempotency key, for the same reason as pages. |
update |
PATCH /api/tasks/<id> with only the changed fields |
the op's own idempotency key: an update whose answer was lost is applied once, never again on top of a later change |
delete |
DELETE /api/tasks/<id> |
idempotent on the server |
- Order. The operations of one task are never reordered: when one does not go through, the later ones of that task wait; other
tasks go on. A task anchored to a page that is still to be created waits for the page (same
sync()call). - Two devices. Updates carry fields, not rows: different fields of the same task both win; the same field goes to the last
request to reach the server. A deletion wins over an edit made elsewhere (
404on the update → nothing left to do). - Settling. While operations of a task wait, its local row stays as the user left it. When the outbox of the task is empty the
row takes the server's copy: the answer just received, or — when the feed brought a newer revision meanwhile, or the answer may
be a replay — a fresh
GET /api/tasks/<id>. A task acknowledged but not settled (app killed between the two steps, the fresh read failed) is settled at the next cycle. - Errors. Offline, timeouts, 401, 429, the third 5xx: the cycle stops, nothing is lost. A 5xx on one operation: backoff
(5 s … 15 min,
expediteEntityOpsends it).400 bad_page(the page is in the server's trash): backoff, not refused — it goes when the page is back. Any other 4xx / 409: the operation is rejected — kept with its payload and its key, counted inSyncStatus.rejected, never retried by itself, never dropped;TasksRepository.retry / discarddecide. - An operation of a kind or verb this build does not execute is marked
unsupported(kept,SyncStatus.unsupported).SyncEngine.supportedEntityOpslists what runs.
7.4 Canvases
A canvas is a page of type canvas plus its spn-canvas-1 document in the canvases table. pushCanvases:
- the page first (a 3.0 server creates the empty canvas, version 1, with the page);
- a canvas never synced (
version == 0):GET /api/canvases/<id>, merged with the local drawing, sent by the next round; PUT /api/canvases/<id> {data, baseVersion}→200→markCanvasPushed(what was drawn during the request stays dirty);409 conflict(or a feed row for a canvas with local changes) → three-way merge by node / edge id (CanvasDocument.merge, base = the last copy common with the server):- the server already holds the local canvas (a save whose answer was lost) → adopted, no conflict, no extra version;
- clean merge → it replaces the local canvas and is pushed on top of the server version, in the same
sync()call; - merge with conflicts (same card changed differently; edited on one side and deleted on the other — SPEC §7: the edited card
is kept) → nothing is changed on either side: the server copy is stored beside the local canvas
(
recordCanvasConflict), which stays editable and leaves the push list;SyncStatus.canvasConflictscounts it;CanvasRepository.conflict / resolvelet the user decide (mine, theirs, or a merged document). A newer server copy that merges cleanly dissolves the conflict.
404(the page is in the server's trash) /400,413 bad_canvas: the canvas keeps its changes,syncErrorsays why, and it is tried again at the next cycle.
Every write of a merge is a compare-and-set on the canvas editSeq: a merge computed on a copy the user has since changed is
computed again, never written over the newer drawing.
7.5 Guarantees added by 2.0
- A task created offline exists exactly once on the server, under its id, whatever is retried.
- No task operation is dropped: it is acknowledged, waiting, rejected (visible) or unsupported (visible).
- A canvas is never overwritten without
baseVersion; neither the local nor the server copy is replaced by a merge that has conflicts; a card nobody removed never disappears.
Covered by EntityFeedTests (8), TaskSyncTests (12), CanvasSyncTests (9), EntityConvergenceTests (seeded: tasks and
canvases on two devices plus the web, lost answers, 5xx, network loss, restarts mid-queue) and LiveEntitySyncTests against the
scratch backend.
7.6 Reminders
The 1.0 reminder path (store.dirtyReminders() → POST/PATCH/DELETE /api/tasks, server-assigned ids, push only) is unchanged.
Turning it into task sync was examined and not done, because the mapping is not safe: (1) existing reminders are linked to
their task by a server-assigned id (serverTaskId), not by a client id, and the store has no "apply remote reminder"; (2) a
reminder drives a local notification — pulling every dated task of the workspace into reminders would schedule notifications
nobody asked for on that device; (3) a reminder has no status doing, priority or project, so a round trip would lose task
fields. What 2.0 gives instead: the tasks created by reminders come back through the feed like any task (TasksRepository
shows them, with their page anchor), and an app that wants a reminder for a task creates it explicitly. A new app feature should
use TasksRepository for tasks and keep reminders for local alarms.
7.7 Limits of 2.0
- Databases, properties, views, rows, relations, concepts and references are read offline, written online (SPEC §6): the
repositories call the server and update the local copy from the answer; offline they throw
RepositoryError.offline. Nothing of those kinds is ever queued by this build. - Task conflicts are field-level last-writer-wins; there is no "both" copy of a task.
- The answer of
GET /api/syncwithreset: truecarries empty lists per kind (the contract says"entities": {}): harmless. - The fake's
purgekeeps the 2.x cascade rules for pages (the 3.0 server also purges sub-pages trashed with the page and their attachments, API-NEXT §4.15); the engine relies on the tombstones either way. - Canvas page conflicts on the page row (title) follow the page rules; only
datais merged by id.