SPB Git

Repository Public

TEST_REPORT.md 18.7 KB

# Test report — VQuant for macOS 2.0.0 (3) "VQuant Next", 2026-11-20

Branch next (local, not pushed, not tagged). Test backend: the frozen VQuant Next API on M2U64, port 5058 (a copy of production data with every backend 6 route), reached through an SSH tunnel. Production was not used by any test (one unintended launch against production is reported at the end).

Evidence classes used here and in FEATURE_PARITY: T integration test against 5058 · U XCUITest click-through · S seen in a window capture · L exercised live from the running app · P real pointer and key events sent to the running app (cliclick, System Events) and the result observed · unit · — written, not exercised.

# Automated

Suite Command Result
Core unit tests (1.x suites + terminal logic, chart drawings and object ids, lab support, portfolio import, markdown editing, research request and event decoding, layout ↔ workspace mapping, shortcut collisions, notification routes) swift test 130 tests in 21 suites passed (integration suites disabled without their environment), plus the sync-migration suite: 5 passed
Integration against 5058: passkey ceremonies with a software authenticator (enrol by setup link, link spent, sign-in, second device, reauth-gated revocation, wrong origin, sign-out), market data, quant jobs, Quant Lab (19 tools, lifecycle with progress, cancel lab and legacy, failures), Studio (bundle, exports md / zip / tex / pdf, note versions, project memory, quote batching), research workspace (create, multipart upload 201 and 415, inspect, files, sessions, approvals list, sandbox, registry, route preview) Scripts/integration-test.sh 130 tests in 21 suites passed; skipped: the real-analysis test c01 (stream, replay, mid-stream join, cancel — it needs two model calls, beyond the allowance of this work) and the DevSession helper
Kit sync tests after the one kit edit (wholeObjectKinds) swift test --filter Sync in VQuantKit 51 passed
UI tests, each class run serially on a quiet screen (final run) VQ_UI_ONLY=<Class> Scripts/ui-test.sh Last run (kit 0.4, after the window-probe fix): 28 tests: 23 passed, 2 failed, 3 skipped (with stated reasons); three of the passes contain an expected failure on a synthesized drag — table below. The run before it: 22 passed, 3 failed. ResearchLiveUITests (2 tests) never run: they call a model
Signed bundle Scripts/verify-release.sh 13 checks passed (Developer ID, hardened runtime, timestamp, sandbox, associated domain, no get-task-allow, embedded profile, no web engine, no web resource, no secret)
Headers Scripts/check-headers.sh ok

# UI tests, final serial run

Class Passed Notes
VQuantUITests (2) 2 signed-out screen and setup-link validation; all 14 sidebar destinations by click, a computation row, a dataset
TerminalUITests (5) 5 overview index → Charting; watchlist click / ⌘ / ⇧ selection, create and delete a list; add and remove symbols; ⌘K → type a ticker → Return; statements tab, common size, inspector, compare. The drag-reorder step is an expected failure (see "synthesized drags")
ResearchUITests (3) 3 Studies scope and the six inspector tabs; stored conversation and trace sheet; model picker "Auto" and Research mode
PortfolioUITests (1) 1 portfolio row, new portfolio, add / edit / ⌘Z / remove a position, Analysis and Rebalancing tabs, delete
StudioUITests (1) 1 create project, open, new note, type, preview, delete
ChartUITests (4) 4 interval and chart type by click, horizontal line by click, undo / redo, provenance popover; ask lands in Research; ask from a detached chart window reaches a main window. Trend-line drag and ticker drop are expected failures (synthesized drags)
LabUITests (3) 2 run an engine from the catalogue; progress and cancel. Failed: testSelectAFinishedJobAndCompare, at its last step (the click on the review button is not delivered after the project menu closes); row selection and compare pass inside it in the last run and work with real clicks (P, capture), and the review sheet opens with a real click (P, capture)
MacUITests (9) 5 passed, 3 skipped, 1 failed passed: Settings panes by click, memory create / pin / delete, Research and Lab menus + View menu, notification route, and — since the window-probe fix — saved layout in the menus, re-open, delete and ⌘K. Skipped with a stated reason: both menu-bar-extra tests (the status item is hidden on this Mac's full menu bar), window restoration (done by Scripts/restore-check.sh). Failed: routing strategy (an XCUITest click on the segmented control is not delivered; with real clicks every click wrote the preference and refreshed the preview in < 0.25 s — P)

Synthesized drags. XCUITest's press(…thenDragTo:) does not start a SwiftUI / AppKit drag session on macOS 27. The three drag paths were therefore checked with real pointer events (P): a watchlist row dragged below another one (order changed on screen and on the server, then restored); a trend line drawn by dragging in the plot and a horizontal line by clicking (capture build/shots/next/chart-drawn-by-pointer.png); a ticker dragged from a watchlist row onto a detached chart window (its title went from AAPL to MSFT).

Clicks not delivered by XCUITest were seen on two custom controls (the kit's segmented control while its thumb animates, a toolbar button right after a menu closed). Real clicks on the same controls always worked. This is reported as a test-harness limit, not hidden: the two failing tests stay failing in the suite.

# Live runs on 5058 (model calls: within the allowance of two fast analyses and one research study)

Run Generation Model Usage reported by the server Seen
Fast, auto fe5fc3c4-7899-499f-8406-93f420f062c1 routed to GPT-5 Nano (economy strategy at that moment) 61.5 k in · 826 out · $0.0034 model_routed line with its reason, usage line, provenance, trace
Research study, cap 3 USD, with mac-test-closes.csv (400 rows) b6a72f8d-a9f6-4889-925e-362c4afc9354 Sonnet 5.5 (7 lead calls) + Haiku 5.5 (1) 529 815 in · 3 385 out · $0.318 upload and inspection card, cockpit while running, 4 code cells (one failed, kept with its traceback), 1 experiment, 2 figures with linter verdicts, quality gate, 13 files; reopened later from REST

Both were started through the composer's own send path by a DEBUG launch argument, not by a click on Send. The second fast analysis of the allowance was not used. Not observed in these runs: an approval request, a background run, a package installation, a budget event, a plan.

Quant engines (no model cost) were run freely: volatility, risk, regression, options (pricer, strategy, chain), backtest with walk-forward, portfolio, legacy tools, a validation failure, cancelled jobs.

# Checked with real pointer and key events (P)

  • Routing strategy clicks in Settings: preference written and router preview refreshed after each of four clicks.
  • "Request review" (then named "Faire vérifier") opens the sheet labelled "Automated review — not a human peer review" (nothing was started).
  • Drag-reorder in a watchlist; drawing by drag and by click on the chart; ticker drop on a detached chart.
  • ⇧⌘N opens a detached chart window with its content.
  • Saved window layout, second round (real menu clicks through System Events, name set in the dialog, 5058): before the fix the toast read "No window to save."; after it, the same sequence wrote the workspace object (one chart panel, config.symbol MSFT, the Mac part in config.mac), listed the layout in the Window menu, and choosing it closed the window and opened a new one titled "Charting – MSFT" with its content (build/shots/next/layout-reopened.png). The test object was deleted afterwards.
  • Quant Lab, second round: four real clicks on rows of the Computations list changed the selection four times; in the two cases where the clicked row could be matched without ambiguity (start time and duration) the highlighted row, the job line and the result agree. One earlier click selected another row than the one aimed at: a fixed coordinate was used on a list at another scroll position — not accounted for further. The Compare menu listed only other results of the same engine, without the current one; choosing one showed the comparison with the right A and B (build/shots/next/lab-compare-by-pointer.png).
  • Window restoration: Scripts/restore-check.sh — three main windows on three sections (one opened with ⌘N and moved to Statements) and three chart windows; real quit keeping windows; relaunch: the six windows returned with the same titles and frames.

# Seen running (window captures, light and dark)

build/shots/next/ (merged tree): overview (light, dark), research, Quant Lab catalogue, chart, chart drawn by pointer, watchlists, chart window after drop. The engineers' captures of every screen are in ~/Library/Developer/VQuant/shots/{terminal,research,lab,chart,portfolio-studio,mac}/ (outside the repository): statements, palette, the six inspector tabs on a recorded and on the live study, approvals sheet (recorded event), trace sheet, model picker, dataset card, lab forms / running / results of seven engines / compare / options / backtest / review, portfolios (valuation, import, analysis, rebalancing proposal), studio (project, notes and versions, reviews, memory, export), the 20 Settings panes, the menu-bar panel in a preview window. Not every capture of every pair was opened; those opened on the merged tree are the ones listed first.

# Language check on VQuantKit 0.4 (second round)

The app builds against VQuantKit 0.4 (ccade44) and asks the kit for English. Seen on that build (build/shots/next/cockpit-kit04-dark.png, lab-result-kit04-light.png): the kit's own vocabulary is English (objective, quality gate, timeline, code cells, lab result). A single language was not reached at first, for the reasons below; 1 and 2 are corrected, 3 is intended:

  1. The six inspector tab titles and the action "Faire vérifier" were French strings of this app (the first brief named them so). The app being English-only, they now read Activity · Code · Experiments · Figures · Files · Environment and "Request review" (the sheet keeps its label "Automated review — not a human peer review"). No other French literal remains in the app sources (searched for accented literals).
  2. The unit of a number of days was printed "j" whatever the language (hard-coded in the kit's formatter). The kit's VQFormat.byUnit now takes the language (default: the host app's) and reads the suffix from its string catalogue: "d" in English, "j" in French; kit tests 163 passed, including the web golden vectors (French) and a new test of both suffixes. That kit edit is left uncommitted in the kit repository for the lead to commit.
  3. Not a language: numbers follow the account's synced number format (fr-CA on the owner's account). Intended.

# Defects found and fixed during this work

  1. A detached chart window opened while signed in was blank (present since 1.0; 1.0.1's report says detached windows were never exercised). Cause: the window's task hung on an empty Group. Same cause for a main window re-opened by a saved layout. Found with real pointer events and the layout UI test.
  2. Restored windows came back on the default section: scene storage can arrive after the first appearance and was only read once. Now applied when it changes, and written for windows that never navigated.
  3. ⌘K opened the palette without keyboard focus in its field (UI test).
  4. VQEmptyState inside a flexible stack pushed the window content off the top (UI test) — empty states are overlays.
  5. /api/quant/jobs failed to decode as soon as one job was cancelled (new status of backend 6) — tolerant decoding.
  6. 1.0.1's ⌃⌘S (Save Window Layout) collided with the system's Show/Hide Sidebar → ⇧⌘S; shortcuts now come from one table with a unit-tested collision check.
  7. HSplitView in the lab workbench looped in AppKit constraints when the comparison opened → own divider.
  8. An accessibility label on selectable text made macOS 27 recurse under XCUITest whenever a conversation was shown.
  9. The watchlist drop indicator stayed after a drop.
  10. A window layout could not always be saved ("No window to save."). The helper that hands a window to SwiftUI code asked for it once, one run-loop turn after its probe view was created; when the view was not attached yet (a main window whose content is inserted after its model exists) the callback never ran: the window was never registered for layouts and WindowModel.window stayed nil. It now reports from viewDidMoveToWindow. Found on the second round with real menu clicks; it also explains why the layout UI test passed in some runs and failed in others.
  11. "1 experiments" in the study header and the inspector summary (counts are now singular or plural).

# Measurements

  • Chart renderer (docs/CHART_PERFORMANCE.md, Debug build, M5 Max, 120 Hz, 1000×640 pt, every bar visible, median main-thread work per frame): 10 000 bars with all indicators 39.4 ms (1.0.1) → 5.0 ms (2.0); 50 000 bars 164.7 → 4.8 ms; 100 000 bars 444.3 → 4.4 ms; with 50 drawings 5.6–6.7 ms. 22 of 23 040 frames over the 8.33 ms budget across 96 scenarios. Metal was not adopted: the time was in the app's own per-frame work, not in rasterisation. Limits: Debug build, other builds running during the measurement, no true 5K display.
  • Memory: the signed Release build, signed in and showing Research, 8 s after launch: 162 MB resident (one ps sample; see the note on that launch below). 1.0.1 reported 142 MB idle. No long-session measurement, no Instruments.
  • Launch time: not measured for 2.0.0 (the one timing taken included the polling tool's own start-up).
  • Bundle: 24 MB universal (1.0.1: 6.9 MB) — the executable now contains the kit and about 19 000 more lines.
  • Sync: with real clicks, a preference write reached the server and the dependent server answer was on screen 60–240 ms after the click (four samples from a temporary log, removed).

# Release

  • Scripts/release.sh: version 2.0.0 (3), universal, signed non-interactively with the existing Developer ID Application identity and the "VQuant Developer ID" profile. Bundle id, entitlements, associated domain and signing settings are unchanged from 1.0.1 (Resources/*.entitlements, Resources/Info.plist untouched).
  • Shipped on 2026-11-20 at the owner's request: notarized by Apple on 2026-11-20 (submissions 7204cfbb… app, 60682e96… disk image, both Accepted), stapled, installed in /Applications and published as VQuant-2.0.0.dmg (sha256 044d90ed…) under spbgit releases vquant-macos/v2.0.0. Gatekeeper: accepted, source=Notarized Developer ID. First launch of the installed app (evidence P, production backend 6): it opened on the owner's existing session without a sign-in prompt and its requests to /api/v2/{models,market/overview,data/quotes,sync/changes,research/*} returned 200. Nothing else of 2.0.0 has been exercised against production.

# One unintended launch against production

To measure launch time and memory, the signed dist/VQuant.app was launched once. Having the production bundle id, it found the owner's existing session in the Keychain and signed in to vquant.spboucher.ai for about ten seconds before being stopped. What it did there: the same read requests a launch of 1.0.1 makes (session, workspace, conversations, indices, quant tools — now with the header vquant-macos/2.0.0) and attempts on /api/v2 routes that production (backend 5.8) does not have. Nothing was written to the server: the sync engine never completed a pull (its replica directory is empty), so no migration and no object write happened. On this Mac it refreshed six files of the encrypted cache and created the empty directory Application Support/VQuant/sync-vquant.spboucher.ai/simon in the app's container. It was not repeated.

# Not verified

  • Touch ID: every ceremony — sign-in on the signed 2.0.0 build by a person, native enrolment, the app lock, revocation and passkey deletion from the Settings panes (the API is tested with a software authenticator).
  • Against production backend 6: nothing — backend 6 is not in production; 2.0.0 has only met the 5058 copy.
  • Research: an approval answered from the Mac and its notification (only a recorded event was rendered); attaching to a study running on another device; background runs, their logs and cancel; budget events; enforced: false banner; the 413 sentence; Quick Look, drag-out to the Finder, Save As in the file browser; deep and ultra modes, Stop and mid-stream join on backend 6; exports, share, image attachment (code carried over from 1.0.1, not re-exercised).
  • Quant Lab: starting a review; add to project; exports and the save panel; "ask the agent"; the limit-of-3 message; the background notification.
  • Studio / Portfolios: a research run filed in a project; drag-reorder of items; "Apply to a Copy"; the file importer and CSV drop; save panels; the main portfolio flows (it is empty on the test copy) and the Risk Analytics refresh after a sync write to main.
  • Terminal: quarterly / balance / cash-flow statement tabs in a capture; CSV export and copy; every context menu; palette routes to notes / portfolios / charts / projects and the recents write.
  • Sync: the migration on a Mac that really holds 1.x watchlists or layouts (the rule is unit-tested against the kit's fake server; the app's own function was never run on such data); a real conflict notice; offline edits with the network physically off; two real Macs.
  • Mac: the real menu-bar item (hidden on this Mac) and its polling cadence; a system notification delivered and clicked (only the router behind it); saved layouts from the menus beyond the UI test that passed once.
  • Offline reading with the network physically off; VoiceOver; Intel Macs; macOS 14–26; a second person's account; long sessions; Instruments.

# Left on the test backend (5058)

Test objects kept so the screens can be reproduced: portfolios mac-test-pf-multi, mac-test-pf-usd; projects mac-test-studio, mac-test-lab-project; notes mac-test-note-1, mac-test-note-2; two project memories; research workspaces named mac-test-… and one "Untitled study" (no delete route); a few dozen quant jobs; the two generations above. preference/default did not exist before this work: it now holds the routing balanced with the three default providers.

Files main
Menu

Theme