SPB Git

Repository Public

9 entries at the root
History

Latest commit feat(api): follow the integration hardening of the server — default error codes, HealthScore.truncated, fake backend idempotent task delete and coded errors, live tests on the new contract; test(document): the ink width law fixture shared with the web 0be0497

NameLast commitUpdated
docs feat(core): additive items asked by the apps — Markdown <u>/<sub>/<sup> like the server (fixture...
Localization test(features): Blocks samples (query answers, transcription stub, source page), catalogue...
Scripts feat(features): Assistant — ask (streaming, scopes, block citations, evidence verdicts,...
Sources feat(api): follow the integration hardening of the server — default error codes,...
Tests feat(api): follow the integration hardening of the server — default error codes,...
.gitignore SPNotesCore 1.0: document model, local store, API client and sync engine shared by SP Notes iOS...
CHANGELOG.md feat(api): follow the integration hardening of the server — default error codes,...
Package.swift feat(features): SPNotesFeatures foundation — router, layout, load states, shared chrome, string...
README.md docs+tests: 2.0.0 — MIGRATION-V2, store/document/API/sync docs, CHANGELOG, server fixtures for...
README.md View source
SP Notes Core logo



SP Notes Core icon

# SP Notes Core

The shared, UI-free Swift core of the native SP Notes apps for iOS, iPadOS and macOS — a lossless ProseMirror document model, a durable local SQLite store, a typed API client with passkeys, and a synchronisation engine whose first rule is that no local modification is ever lost.

Author: Simon-Pierre Boucher

backend version platforms license author

Swift swift-tools-version Swift Package Manager dependencies SQLite AuthenticationServices Keychain concurrency Swift Testing XCTest

commits last commit activity language languages size branches tags clones

modules source files source lines test files test lines tests node types local tables API methods docs

package products default localization consumers isolated builds


# Table of contents


# 🎯 Overview

SPNotesCore is a Swift package with no user interface and no third-party dependency. It contains everything the two native SP Notes apps have in common: how a note is represented, where it is stored on the device, how the app talks to the backend, and how the device and the server are kept in step.

The backend is note-spboucher-ai (production: https://note.spboucher.ai), which stores a note as ProseMirror / Tiptap JSON and exposes a change feed for native clients. The core must stay byte-compatible with it: plain text, word count and Markdown export reproduce the server's own helpers, and every document the core emits is validated against the real web schema.

Two rules run through the whole package:

  • No note is ever lost (store): local edits are never overwritten by the feed; every mutation is one atomic SQLite transaction; nothing waits for the network.
  • No local modification is lost or silently overwritten (sync): concurrent edits are merged when that is safe; otherwise both versions are kept and the user decides.

# Family

Repository Role Platform
note-spboucher-ai Web app, API, database, sync feed Web / PWA — note.spboucher.ai
sp-notes-core (this one) Shared Swift package iOS 18+ · macOS 15+
sp-notes-macos SP Notes for Mac macOS
sp-notes-ios SP Notes for iPhone and iPad iOS / iPadOS

Both apps expect this repository to be cloned next to them as SPNotesCore (they reference ../SPNotesCore).

# ✨ Features

Module What it gives
SPNotesDocument ProseMirror JSON ⇄ a flat list of lines an editor can drive, without losing anything (unknown nodes included); editing commands (blocks, lists, marks, links, tables); plain text, word count, preview, Markdown import/export, HTML export; three-way merge by block id; schema validation
SPNotesStore Durable local database on the system SQLite: pages, tags, attachments, ops queue, conflicts, local versions, reminders, meta; FTS5 search that ignores case and diacritics; change notification streams; multi-process access (app, share extension, read-only widget)
SPNotesAPI The only module that talks to the backend: hardened URLSession transport, hand-written cookies, typed errors, one method per route, multipart uploads, downloads, server-sent events for the AI routes, passkey ceremonies through AuthenticationServices, Keychain token store
SPNotesSync SyncEngine actor: push then pull in bounded rounds, idempotent retries, lost-acknowledgement recovery, three-way merge, conflict records, trash/purge/server-restore handling, background downloads of attachments
SPNotesCore Umbrella module: one import SPNotesCore re-exports the four others

# 🏗 Architecture

flowchart TB
    subgraph APPS["Native apps (own the UI and the attachment directory)"]
        MAC["SP Notes macOS<br/>AppKit · TextKit 2"]
        IOS["SP Notes iOS / iPadOS<br/>SwiftUI · UIKit · TextKit 2"]
    end
    subgraph CORE["SPNotesCore (this package)"]
        SYNC["SPNotesSync<br/>SyncEngine (actor)"]
        DOC["SPNotesDocument<br/>Foundation only · value types"]
        STORE["SPNotesStore<br/>NotesStore (actor) · SQLite WAL · FTS5"]
        API["SPNotesAPI<br/>HTTPClient (actor) · NotesAPI · passkeys"]
    end
    BACK["note-spboucher-ai<br/>https://note.spboucher.ai"]
    DB[("SQLite file<br/>on the device")]
    KC["Keychain<br/>session token"]
    MAC --> CORE
    IOS --> CORE
    SYNC --> DOC
    SYNC --> STORE
    SYNC --> API
    STORE --> DB
    API --> KC
    API -- "HTTPS · cookie written by hand" --> BACK

The three lower modules are independent of one another:

  • SPNotesStore does not import SPNotesDocument: note content is opaque JSON Data, and the caller supplies the derived plain text, word count and preview.
  • SPNotesAPI depends on Foundation, Security and AuthenticationServices only: note content and properties travel as RawJSON and are interpreted by SPNotesDocument.
  • Only SPNotesSync depends on the three others.

# One sync cycle

sequenceDiagram
    autonumber
    participant A as App
    participant E as SyncEngine
    participant S as NotesStore
    participant B as Backend
    A->>S: local mutation (one SQLite transaction)
    A->>E: schedule() — debounced, 2 s by default
    loop up to 4 rounds
        E->>B: POST /api/pages (creations, parents first, client-chosen id)
        E->>B: POST /api/attachments (uploads)
        E->>B: PATCH with baseVersion (title · content · properties)
        alt 409 conflict
            E->>S: reconcile: adopt · move base · three-way merge · or record a conflict
        end
        E->>B: queued actions with their idempotency key, then reminders
        E->>B: GET /api/sync from the stored cursor, until more is false
        E->>S: apply each answer in one transaction with its cursor
    end
    E->>B: download up to 40 wanted files
    E-->>A: statusUpdates() — phase, pending, conflicts, needsReauth

# 🧱 Stack

Layer Technology Version Role
Language Swift, language mode 6 on every target tools 6.0 Strict concurrency: actors, Sendable value types
Package Swift Package Manager — 4 libraries + the umbrella; defaultLocalization: "fr"
Platforms iOS, macOS 18 · 15 (minimum) Deployment targets of the package
Storage system SQLite (linkedLibrary("sqlite3")) — WAL, synchronous=FULL, FTS5
Network Foundation URLSession (ephemeral) — TLS ≥ 1.2, no cookie storage, no cache, redirects refused
Sign-in AuthenticationServices — Platform passkeys
Secrets Security (Keychain) — Session token, this device only, not synchronised
Tests Swift Testing (@Test) and XCTest — 301 + 99 tests
Third-party code none — —

# 🗂 Repository layout

87 tracked files.

text
sp-notes-core/                       # clone it as SPNotesCore next to the apps
├── Package.swift                    # products, SPN_ONLY switch
├── Sources/
│   ├── SPNotesDocument/             # 14 files · 3 467 lines
│   │   ├── Model.swift  JSONValue.swift  Schema.swift  SchemaValidation.swift  BlockIds.swift
│   │   ├── Commands.swift  InlineCommands.swift  Inline.swift  Tables.swift
│   │   ├── Text.swift  MarkdownImport.swift  MarkdownExport.swift  HTMLExport.swift
│   │   └── Merge.swift
│   ├── SPNotesStore/                # 9 files · 2 966 lines
│   │   ├── NotesStore.swift  SQLite.swift  Schema.swift  Models.swift
│   │   └── NotesStore+Pages / +Queries / +Sync / +Queue / +History .swift
│   ├── SPNotesAPI/                  # 14 files · 3 479 lines
│   │   ├── APIConfiguration.swift  HTTPClient.swift  APIError.swift  JSONSupport.swift
│   │   ├── NotesAPI.swift  PageModels.swift  ServiceModels.swift  AIModels.swift
│   │   ├── MultipartForm.swift  SSEParser.swift
│   │   └── AuthService.swift  PasskeyCeremony.swift  PasskeyAuthorizer.swift  SessionTokenStore.swift
│   ├── SPNotesSync/                 # 1 file · 1 014 lines
│   │   └── SyncEngine.swift
│   └── SPNotesCore/                 # umbrella: @_exported import of the four modules
├── Tests/
│   ├── SPNotesDocumentTests/        # 5 suites + Fixtures/ (6 JSON files)
│   ├── SPNotesStoreTests/           # 5 suites
│   ├── SPNotesAPITests/             # 6 suites (URLProtocol stub + live tests)
│   └── SPNotesSyncTests/            # 6 suites + FakeBackend.swift (in-memory backend, 1 072 lines)
└── docs/
    ├── DOCUMENT_API.md  STORE_API.md  API_CLIENT.md  SYNC_ENGINE.md
    ├── reference/                   # read-only copies of the web sources the core must match + BACKEND_CONTRACT.md
    └── brand/

# 🚀 Getting started

# Build and test

sh
swift build
swift test

One module in isolation (parallel development — each with its own scratch directory):

sh
SPN_ONLY=Document swift test --scratch-path .build-document
SPN_ONLY=Store    swift test --scratch-path .build-store
SPN_ONLY=API      swift build --scratch-path .build-api && SPN_ONLY=API swift test --scratch-path .build-api
SPN_ONLY=Sync     swift test --scratch-path .build-sync

SPN_ONLY=Sync also declares the three modules it depends on. The umbrella product SPNotesCore only exists when SPN_ONLY is unset.

# Use it from an app

Clone this repository next to the app as SPNotesCore and reference it as a local package:

swift
.package(path: "../SPNotesCore")
// then depend on the product "SPNotesCore" (everything) or on a single module
swift
import SPNotesCore

let configuration = APIConfiguration.production(appVersion: "1.0.0")      // https://note.spboucher.ai
let tokens = KeychainTokenStore(configuration: configuration, accessGroup: "<team>.<keychain group>")
let client = HTTPClient(configuration: configuration, tokenStore: tokens)
let store  = try NotesStore(url: databaseURL)
let engine = SyncEngine(store: store, api: NotesAPI(client: client), files: locator, options: SyncOptions())

var doc = try NoteDocument(json: data)          // ProseMirror JSON → lines
doc.toggleMark(.bold, in: 0..<5, at: 2)         // edit
doc.ensureBlockIds()                            // before saving: merges match blocks by id
let body = try doc.jsonData()                   // lines → canonical JSON (sorted keys)

await engine.schedule()                         // after every local mutation
await engine.sync()                             // app start, foreground, network back, pull-to-refresh

# ⚙️ Configuration

There is no configuration file. Behaviour is set by types and by a few environment variables used at build or test time.

Name Where Purpose
SPN_ONLY Package.swift Document, Store, API or Sync: build and test a single module
SPN_LIVE_URL, SPN_LIVE_TOKEN API and sync tests enable the live tests against the test server; any *.spboucher.ai host is refused
SPN_LIVE_TOKEN_DISPOSABLE, SPN_LIVE_TOKEN_DISPOSABLE2 API tests optional: two more tokens (one is revoked, one logs out)
SPN_RANDOM_SEEDS, SPN_RANDOM_ONLY sync tests extra seeds for the randomised convergence test, or a single seed to replay
SPN_DUMP document tests enables the dump test
Type Settings
APIConfiguration baseURL, userAgent, requestTimeout, resourceTimeout, streamIdleTimeout, sessionCookieName, ceremonyCookieName; origin and relyingPartyID are derived from the base URL
SyncOptions autoDownloadLimit (25 MiB), includeDatabaseRows (false), feedPageSize (200)
KeychainTokenStore service, account (defaults to the origin, so a test server never shares a token with production), accessGroup, useDataProtectionKeychain

# 📝 Module: SPNotesDocument

Foundation only, value types, all Sendable and Hashable. Full reference: docs/DOCUMENT_API.md.

Area API (excerpt)
Model JSONValue, PMNode, PMMark, NoteDocument (lines), Line (displayString, inlineLength)
Reading / writing NoteDocument(json:), jsonData() (canonical JSON, sorted keys), ensureBlockIds(), isStructurallyEqual(to:)
Commands public mutating func on NoteDocument; they return whether the document changed, never trap on out-of-range input, and leave a document valid for the web schema
Marks MarkKind: bold, italic, underline, strike, code, highlight, subscript, superscript; links through setLink / removeLink
Lists ListKind: bullet, ordered, task
Text plainText(), wordCount, preview(maxLength:), pageReferences(), attachmentReferences(), taskItems()
Export / import markdown(), html(title:), htmlBody(), Markdown import
Merge NoteDocument.merge(base:mine:theirs:) → .merged(doc) or .conflict
Validation schemaViolations(strict:)
The 30 node types of the web schema known to the core

doc, text, paragraph, heading, codeBlock, footnote, detailsSummary, bulletList, orderedList, taskList, listItem, taskItem, blockquote, callout, aiBlock, details, detailsContent, table, tableRow, tableHeader, tableCell, image, fileBlock, blockMath, horizontalRule, youtube, hardBreak, mention, inlineMath, footnoteRef.

Unknown nodes are kept as they are and written back unchanged.

Conventions worth knowing before writing an editor on top of it:

  • Inline offsets are UTF-16 code units (NSString / NSRange compatible). An inline atom (hard break, mention, inline math, footnote reference, unknown inline node) counts as exactly 1 unit: display it as one U+FFFC attachment character.
  • New nodes are created with every schema attribute (defaults included) and a fresh block id, exactly as the web editor serializes them.
  • plainText, wordCount and markdown reproduce the server byte for byte (checked on the fixtures and on 400 random trees against outputs recorded from the web application's own helpers).
  • Three-way merge: top-level blocks are matched by block id. A block changed, inserted, deleted or moved on one side only is taken from that side; the merge never drops a change silently. A top-level block is the whole node (an entire list, quote or table): two edits inside the same list conflict.
  • HTML export uses the same markup as the server's print view, without KaTeX — math is exported as its LaTeX source.

# 🗄 Module: SPNotesStore

The durable local database. Full reference: docs/STORE_API.md.

swift
public actor NotesStore {
    public init(url: URL, readOnly: Bool = false) throws
    public static func inMemory() throws -> NotesStore
    public func integrityCheck() throws -> [String]     // empty = healthy
    public func checkpoint() throws                     // wal_checkpoint(TRUNCATE)
    public func rebuildSearchIndex() throws
    public func wipeAll() throws                        // "erase this device" only
}
Aspect Behaviour
Connection SQLITE_OPEN_FULLMUTEX, journal_mode=WAL, synchronous=FULL, foreign_keys=ON, busy_timeout 8 s
Transactions every mutation is ONE BEGIN IMMEDIATE transaction; an error rolls it back entirely (index, tags, queue and cursor included) and publishes no change event
Multi-process the same file may be opened by the app (read/write), a share extension (read/write) and a widget (readOnly: true); a read-only store never creates nor migrates the file
Schema PRAGMA user_version, one transaction per migration step; a file newer than the build is refused (StoreError.schema)
Ids and dates lowercase UUID strings; timestamps stored as integer milliseconds since the Unix epoch
Dirty state needsCreate, dirtyContent, dirtyMeta (parent, section, position, pinned, archived, icon, cover, inbox), tagsDirty, editSeq (+1 on every local change to push)
Base copy PageBase { title, content, properties, version }: the last synced copy of the version-guarded fields, captured at the first content edit and dropped when the content is clean again
Sync state conflict · error · pending · synced
Search FTS5 (title, body, tags), tokenizer unicode61 remove_diacritics 2; every term must match, the last one as a prefix; terms are quoted so no input can cause a syntax error; bm25 weights title 10, tags 4, body 1; incremental index
Ops queue trash, restore, purge, deleteAttachment, createTask, updateTask, deleteTask, triage; leased for 120 s; per-target order never broken; backoff 5 s, 10 s, 20 s … capped at 15 min; one idempotency key per op, never changed across retries
Change notification changes() -> AsyncStream<StoreChange>: pages, tags, attachments, queue, conflicts, reminders, meta, reset; emitted after the commit; noteExternalChange() when another process wrote the file
Errors StoreError: sqlite, notFound, invalid, cycle, readOnly, closed, schema

# 🌐 Module: SPNotesAPI

The only module that talks to the backend. Full reference: docs/API_CLIENT.md; contract: docs/reference/BACKEND_CONTRACT.md.

Type Role
APIConfiguration base URL, Origin, User-Agent, timeouts, cookie names
SessionTokenStore, KeychainTokenStore, InMemoryTokenStore where the session token lives
HTTPClient (actor) transport: cookies by hand, headers, uploads, downloads, event streams, session events
APIError every failure, typed
NotesAPI one method per route
PasskeyCeremony, PasskeyAuthorizing, SystemPasskeyAuthorizer, AuthService passkeys and sign-in flows
SSEParser, AIEvent server-sent events of the AI routes

Transport rules implemented by HTTPClient:

  • One dedicated ephemeral URLSession: no cookie storage, no URL cache, no credential storage, TLS ≥ 1.2. Redirects are refused (the cookie never follows one).
  • Cookie: __Host-note_session=<token> is written by hand from the token store; a Set-Cookie on any answer (sign-in, step-up rotation) replaces the token and saves it.
  • A cleared cookie, or a 401 answering a request that carried the current token, clears the store and emits SessionEvent.signedOut exactly once (a late 401 for a token that has since been rotated is ignored).
  • The WebAuthn ceremony cookie is captured from the options call and sent only on the matching verify call, once.
  • Origin and Idempotency-Key only on POST / PATCH / PUT / DELETE; x-request-id on every request.
  • JSON bodies are encoded with sorted keys (same request → same bytes); answers are decoded with ISO 8601 dates with or without fractional seconds, and the PostgreSQL text form that /api/search returns.
  • User-Agent SPNotes/<x.y.z> (iPhone|iPad|Macintosh; <os>) native → the server labels the session "SP Notes · iPhone|iPad|Mac".

Routes covered by NotesAPI:

Group Routes
Change feed GET /api/sync?since=&limit=
Pages GET/POST /api/pages, GET/PATCH/DELETE/POST /api/pages/<id>, resolve, versions, tags
Attachments POST/GET/DELETE /api/attachments, GET /api/files/<id>
Authentication login, reauth (step-up), passkeys, sessions, logout, state
AI chat and inline actions as server-sent events (data: <json>, terminator data: [DONE])
Other search, tags, tasks (reminders), trash, inbox and triage, capture, health

# 🔄 Module: SPNotesSync

SyncEngine is an actor that keeps one NotesStore in step with the backend through NotesAPI. Full reference: docs/SYNC_ENGINE.md.

# What the apps call, and when

Moment Call
after every local mutation of the store (edit, tag, pin, move, trash, attachment, reminder…) await engine.schedule() (debounced, 2 s by default) and, for the badge, await engine.refreshCounts()
app start, return to foreground, network back, pull-to-refresh, push notification, after sign-in await engine.sync()
after a successful passkey step-up, when status.needsReauth was true await engine.sync() (the queued permanent deletions go at once)
the user opens a file that is not on the device try await engine.download(attachmentId:)
the user settles a conflict try await engine.resolveConflict(pageId:choice:)
sign-out, app about to be suspended for good await engine.stop()
UI state (spinner, offline banner, counters) for await status in await engine.statusUpdates()

sync() returns .completed, .offline, .signedOut or .failed(message); nothing is lost in any of these cases. It is re-entrant: a call made while a cycle runs waits for the next cycle, so its outcome always covers the state of the store at the time of the call.

# Guarantees

  1. A local edit is acknowledged only for the exact state that was sent (editSeq): what is typed while a request is in flight stays dirty and goes with the next round or cycle.
  2. Local text is replaced in exactly three cases, and each time the replaced text is first saved as a local version: an automatic merge, the user choosing "theirs" or "both", a server copy identical to the local one.
  3. The server's text is never overwritten without baseVersion, and only on top of a copy the local text derives from.
  4. A page purged on the server while this device holds unsynced text is created again under the same id.
  5. Nothing is duplicated by retries (page, file, task, conflict copy).
  6. Killing the app at any point loses nothing: all state is in the store.

# Idempotency

Request What makes a retry safe
create page, upload attachment the client-chosen id (200 + x-existing) — deliberately no idempotency key
guarded PATCH baseVersion; a retry gets 409 with a server copy equal to the local one → adopted
unguarded PATCH, PUT tags idempotent by nature
trash, restore, delete attachment, triage the op's own key (one key per op, never reused for another payload)
create task spn-task-<reminder id>
resolve conflict spn-res-<page>-<server version>-<choice>-<fingerprint of the local text>

# Conflict policy

Reconciliation happens when a guarded write is refused (409) or when the feed brings a newer version of a note with unsynced text:

  1. server copy = local copy → adopted, nothing to push;
  2. server copy = what this device sent before a lost answer → base moved, local text pushed next;
  3. server version lower than the local base (restored server) → the local text is sent again on top, with a snapshot;
  4. three-way merge with the stored base copy: title (one side changed), content (top-level blocks by id), properties (key by key);
  5. otherwise a conflict is recorded: the local text is untouched and stays editable, the server copy is stored beside it, the note leaves the push list until the user decides.
Choice Note on the server Where the other text survives
.keepMine local title and content server history
.keepTheirs unchanged server history (not applied) and local versions
.keepBoth unchanged; the local text becomes a sibling page "… (conflict copy)" both are notes

Flags, tags, moves, icon, cover and position are not versioned: last writer wins, they never conflict.

Error classification, trash, purge and server restore
  • Stops the cycle: 401 (.signedOut); no connectivity, connection failures, timeouts (.offline); 429, the third 5xx of the cycle, a feed that breaks the contract, a closed store (.failed); cancellation (.offline).
  • Concerns one item: everything else. A page gets syncError; an op gets the store's backoff; an upload refused with a 4xx is not sent again before a backoff.
  • 403 reauth_required on a purge is not a failure: the op stays first in line without counting an attempt and needsReauth stays true.
  • Loops are bounded: 4 rounds per cycle; the feed must advance, never go backwards, and may ask for a reset twice at most.
  • Trash is not a conflict: if a note is trashed elsewhere while this device edits it, the edit stays pending; restoring the note lets the edit through.
  • Purge on the server: a clean page disappears; a page with unsynced text is created again with the same id.
  • Purge on the device: queued; needs a fresh passkey verification.
  • reset feed (the server database was restored): everything is downloaded again; dirty notes are kept and pushed; synced notes the restored server no longer has are created again on it. Nothing is deleted on the device in this mode.

# 🗃 Local data model

Schema version 1 (Sources/SPNotesStore/Schema.swift): 10 tables and one FTS5 virtual table.

Table Content
pages the notes: server fields, opaque content and properties, dirty flags, edit_seq, sync state, plain text and preview
page_bases last synced copy of the version-guarded fields of a page being edited
tags, page_tags tag names and their pages
pages_fts FTS5 index (title, body, tags)
attachments file metadata, upload state, local path, last access
ops queue of actions that are not dirty state, with their idempotency key and backoff
conflicts the server copy stored beside a note in conflict
local_versions local history, including every text replaced by a merge or a resolution
reminders reminders (pushed to the server as tasks)
meta key/value: feed cursor, sync.sent.*, sync.fullCopyPending…

Logical relations (most tables are keyed by the page id rather than by a declared foreign key):

erDiagram
    pages ||--o| page_bases : "base copy"
    pages ||--o{ page_tags : tagged
    pages ||--o{ attachments : holds
    pages ||--o| conflicts : "server copy"
    pages ||--o{ local_versions : history
    pages ||--o{ reminders : has
    pages ||--|| pages_fts : indexed

Data conventions shared by the native apps: favourite = properties.favorite === true; folder = a page with properties.folder === true; reminder = a task { pageId, dueAt }; a PencilKit drawing = an image node whose alt is drawing-<uuid>.png plus a companion attachment drawing-<uuid>.pkdrawing on the same page (editable source).

# 🔒 Security and privacy

  • The session token lives in the Keychain (kSecAttrAccessibleAfterFirstUnlockThisDeviceOnly, not synchronised). It is never logged; APIError descriptions never contain request data.
  • The Keychain account defaults to the origin, so a test server never shares a token with production.
  • The URLSession is ephemeral, stores no cookie, no cache and no credential, requires TLS ≥ 1.2 and refuses redirects.
  • Mutating requests carry the exact Origin the backend expects; permanent deletions need a passkey verification younger than 5 minutes (reauth_required).
  • Passkeys go through the system (AuthenticationServices); PasskeyCeremony only does the JSON work.
  • Link and image URLs of the HTML export go through the same allow-list as the server; javascript: links are dropped by the Markdown import.
  • The live tests refuse any *.spboucher.ai host and clean up what they create.

# 🧪 Testing

400 tests (2.0.0, branch next): 301 with Swift Testing (@Test) and 99 with XCTest (the API module; the 11 live tests are skipped without SPN_LIVE_URL). Per module: Document 131, Store 82, API 99, Sync 67, Design 21. The table below lists the 1.0 suites; 2.0 adds NextNodeExportTests (9, one of them over the 4 fixtures published by the backend) and NextNodeModelTests (13), InkTests (12), CanvasModelTests (4), CanvasMergeTests (8), CanvasLayoutTests (4) in Document; MigrationV2Tests (17), EntityTests (15) and the fixture generator (1, inert without SPN_MAKE_V1_FIXTURE) in Store; RetryTests (11) in API; NextNodesSyncTests (1) and one more queue test in Sync; SPNotesDesignTests (21). What 2.0 changes is described in CHANGELOG.md, docs/MIGRATION-V2.md and docs/DESIGN.md.

Module Suite Tests
Document (81) CommandTests 35
ModelTests 22
TextTests 16
MergeTests 7
DumpTests 1
Store (49) SyncTests 15
PageTests 12
EngineTests 11
SearchTests 6
QueueHistoryTests 5
API (88) HTTPClientTests 22
ErrorAndModelTests 17
PasskeyAndAuthTests 15
RoutesAndMultipartTests 12
SSEAndAITests 11
LiveBackendTests (only with SPN_LIVE_URL) 11
Sync (65) FaultTests 26
MergeConflictTests 16
TrashAttachmentQueueTests 14
SyncEngineTests 7
RandomConvergenceTests (seeded) 1
LiveSmokeTests (only with the live variables) 1
  • Document: fixtures recorded from the web application (Tests/SPNotesDocumentTests/Fixtures); swift test writes every document the module emits to /tmp/spn-docs.json so they can be validated against the real web schema.
  • API: a URLProtocol stub (Tests/SPNotesAPITests/TestSupport.swift).
  • Sync: an in-memory fake of the backend written from the server's route handlers (Tests/SPNotesSyncTests/FakeBackend.swift), real store files, 2–3 devices, fault injection, and a seeded randomised convergence test.

Live tests against the test server (never production):

sh
ssh -f -N -L 8482:localhost:8482 M4M64c
TOKEN=$(ssh M4M64c 'bash -lc "cd ~/dev/note-native && set -a && . ./.env && set +a && node_modules/.bin/tsx scripts/dev-session.ts"')
SPN_LIVE_URL=http://localhost:8482 SPN_LIVE_TOKEN=$TOKEN SPN_ONLY=API swift test --scratch-path .build-api

A development token is "fresh" for 5 minutes: within that window the purge and revocation paths run for real, afterwards the tests check the typed reauth_required error instead.

Validating emitted documents against the real web schema (the dev worktree on the backend node, never production):

sh
scp /tmp/spn-docs.json M4M64c:/tmp/spn-docs.json
ssh M4M64c 'bash -lc "cd ~/dev/note-v2 && node_modules/.bin/tsx .spn-schema.mts validate /tmp/spn-docs.json"'

Every line must read ok … identical.

# 🧯 Troubleshooting and gotchas

Trap Explanation
Never send an idempotency key with a page creation or an upload The server replays the stored answer of the first attempt for 48 h, whatever the new body, even if the page was purged since. The client-chosen id is what makes the retry safe.
The answer of a creation is never trusted blindly The returned row is always compared with the local one.
Call ensureBlockIds() before saving Merges match blocks by id. Notes without ids still work but merge less well.
KeychainTokenStore fails in an unsigned tool on macOS The data-protection keychain needs a signed app with a keychain access group; pass useDataProtectionKeychain: false.
A widget cannot open the store A read-only store never creates nor migrates the file, and the directory must be writable by the reader (SQLite needs the -shm file).
The share extension wrote but the app shows nothing Only changes made through the same NotesStore instance are observed: signal the app yourself (Darwin notification) and call noteExternalChange().
After markAttachmentDeleted(id:) The app deletes the returned file itself: attachment files belong to the caller.
removeSyncedPages(notIn:) Scans every clean page: use it only after a full re-download.
== on NoteDocument Compares container keys too; use isStructurallyEqual(to:) to compare content only.
Two parallel module builds Give each its own --scratch-path.
Test server Do not modify its worktree and do not reset its database.
Differences observed between the contract and the test server (2026-11-18)
  • GET /api/sync with a cursor ahead of the server answers {reset: true, cursor: 0, more: true, …} without user. limit bounds each list separately. A non-numeric or negative since → 400 bad_cursor.
  • GET /api/search?mode=quick returns updatedAt in PostgreSQL text form, not ISO 8601.
  • Page rows of POST/GET /api/pages… carry workspaceId, plainText, tsv and no tags (the feed has them).
  • POST /api/pages/<id>?action=duplicate answers 200 (not 201).
  • Task priority is low | medium | high (400 otherwise).
  • POST /api/inbox/<id> also returns created; POST /api/inbox/undo returns {ok, already}; triage actions are keep | task | database | reference | archive | trash.
  • DELETE /api/auth/sessions/<id> needs a fresh verification (403 reauth_required), like the global revocation.
  • Idempotency: a replayed answer has idempotent-replay: true but not x-existing; a key reused with a different body on the same method and path is replayed as well.
  • Extra error codes: bad_request, challenge_missing, no_instruction / bad_language, last_passkey (409), bad_cursor. not_enrolled is a 409. Validation errors are 400 without a code. A malformed UUID in a path answers 500.
  • GET /api/health is public and does not echo x-request-id.
  • GET /api/tags reports pages: 0 for tags carried by live pages.
  • DELETE /api/attachments?id= is a soft delete: /api/files/<id> keeps serving the bytes. Purging a page leaves its attachments alive with pageId: null (no tombstone).

# 🚧 Known limitations

Documented in the module references.

Sync (docs/SYNC_ENGINE.md)

  • Reminders are pushed, not pulled: tasks changed or created on the web do not come back.
  • Folders purged on the device: the store queues one purge for the root and the server's purge does not cascade, so the notes inside stay on the server.
  • Moves are last-writer-wins and the server does not check cycles.
  • The store has no compare-and-set on its conflict helpers: an edit landing in the few microseconds between the read and the write of a merge is replaced by the merge (it is kept as a local version, not lost).
  • Undetectable restore: if a restored server has already received enough new writes for its head to pass a device's cursor, no reset is sent.
  • A page that turns into a database row on the web stops being updated on the device (unless includeDatabaseRows).
  • Pending pages are pushed 500 per round, 4 rounds per cycle; a larger backlog needs another sync().
  • Ops refused for good by the server are retried with backoff for ever rather than dropped.
  • An edit of the title only, or of the same block, on two devices is a conflict by design.
  • SyncOutcome has no "cancelled" case: stop() reads as .offline.

Store (docs/STORE_API.md)

  • trash / restore cancellation ignores pages moved into or out of the subtree between the two calls.
  • Properties equality ignores key order but is otherwise structural; content equality is byte equality.

Document (docs/DOCUMENT_API.md)

  • Two documents differing only by Unicode normalization are ==.
  • A lone UTF-16 surrogate escape in JSON is read as U+FFFD.
  • Marks are toggled on text only; inline atoms keep their own marks.
  • Table helpers address cells by their position in each row; they do not resolve colspan / rowspan geometry.
  • Reference-style links, footnotes and HTML blocks are imported from Markdown as text.

# 📚 Documentation

Document Content
docs/DOCUMENT_API.md public API of SPNotesDocument: conventions, model, commands, tables, text/Markdown/HTML, merge, validation
docs/STORE_API.md public API and exact semantics of SPNotesStore (12 sections)
docs/API_CLIENT.md SPNotesAPI: set-up, transport rules, errors, endpoints, passkeys, server-sent events, tests
docs/SYNC_ENGINE.md the cycle, its guarantees, the conflict policy, what the apps must call, known limits
docs/reference/BACKEND_CONTRACT.md the backend contract used by the native apps
docs/reference/README.md read-only copies of the web sources the core must stay compatible with (text.ts, markdown.ts, html.ts, validate.ts, same-json.ts, service.ts, web-editor-schema.txt)

# 👤 Author

Simon-Pierre Boucher — design, development, operations

www.spboucher.ai · git.spboucher.ai · contact@spboucher.ai

# 📄 License

All rights reserved © 2026 Simon-Pierre Boucher.

Clone sp-notes-core

Anonymous clone and fetch. Pushing requires the owner's personal access token.

Snapshot of main

Files main
Menu

Theme