SPB Git

Repository Public

TEST_REPORT.md 18.7 KB 198 lines markdown

1<!-- Author: Simon-Pierre Boucher — contact@spboucher.ai · File: docs/TEST_REPORT.md -->2# Test report — VQuant for macOS 2.0.0 (3) "VQuant Next", 2026-11-2034Branch `next` (local, not pushed, not tagged). Test backend: the frozen VQuant Next API on M2U64, port 5058 (a copy5of production data with every backend 6 route), reached through an SSH tunnel. Production was not used by any test6(one unintended launch against production is reported at the end).78Evidence classes used here and in FEATURE_PARITY: **T** integration test against 5058 · **U** XCUITest click-through ·9**S** seen in a window capture · **L** exercised live from the running app · **P** real pointer and key events sent10to the running app (`cliclick`, System Events) and the result observed · **unit** · **—** written, not exercised.1112## Automated1314| Suite | Command | Result |15|---|---|---|16| Core unit tests (1.x suites + terminal logic, chart drawings and object ids, lab support, portfolio import, markdown editing, research request and event decoding, layout ↔ workspace mapping, shortcut collisions, notification routes) | `swift test` | **130 tests in 21 suites passed** (integration suites disabled without their environment), plus the sync-migration suite: **5 passed** |17| Integration against 5058: passkey ceremonies with a software authenticator (enrol by setup link, link spent, sign-in, second device, reauth-gated revocation, wrong origin, sign-out), market data, quant jobs, Quant Lab (19 tools, lifecycle with progress, cancel lab and legacy, failures), Studio (bundle, exports md / zip / tex / pdf, note versions, project memory, quote batching), research workspace (create, multipart upload 201 and 415, inspect, files, sessions, approvals list, sandbox, registry, route preview) | `Scripts/integration-test.sh` | **130 tests in 21 suites passed**; skipped: the real-analysis test `c01` (stream, replay, mid-stream join, cancel — it needs two model calls, beyond the allowance of this work) and the DevSession helper |18| Kit sync tests after the one kit edit (`wholeObjectKinds`) | `swift test --filter Sync` in VQuantKit | 51 passed |19| UI tests, each class run serially on a quiet screen (final run) | `VQ_UI_ONLY=<Class> Scripts/ui-test.sh` | Last run (kit 0.4, after the window-probe fix): 28 tests: **23 passed, 2 failed, 3 skipped** (with stated reasons); three of the passes contain an expected failure on a synthesized drag — table below. The run before it: 22 passed, 3 failed. `ResearchLiveUITests` (2 tests) never run: they call a model |20| Signed bundle | `Scripts/verify-release.sh` | 13 checks passed (Developer ID, hardened runtime, timestamp, sandbox, associated domain, no get-task-allow, embedded profile, no web engine, no web resource, no secret) |21| Headers | `Scripts/check-headers.sh` | ok |2223### UI tests, final serial run2425| Class | Passed | Notes |26|---|---|---|27| `VQuantUITests` (2) | 2 | signed-out screen and setup-link validation; all 14 sidebar destinations by click, a computation row, a dataset |28| `TerminalUITests` (5) | 5 | overview index → Charting; watchlist click / ⌘ / ⇧ selection, create and delete a list; add and remove symbols; ⌘K → type a ticker → Return; statements tab, common size, inspector, compare. The drag-reorder step is an **expected failure** (see "synthesized drags") |29| `ResearchUITests` (3) | 3 | Studies scope and the six inspector tabs; stored conversation and trace sheet; model picker "Auto" and Research mode |30| `PortfolioUITests` (1) | 1 | portfolio row, new portfolio, add / edit / ⌘Z / remove a position, Analysis and Rebalancing tabs, delete |31| `StudioUITests` (1) | 1 | create project, open, new note, type, preview, delete |32| `ChartUITests` (4) | 4 | interval and chart type by click, horizontal line by click, undo / redo, provenance popover; ask lands in Research; ask from a detached chart window reaches a main window. Trend-line drag and ticker drop are **expected failures** (synthesized drags) |33| `LabUITests` (3) | 2 | run an engine from the catalogue; progress and cancel. **Failed**: `testSelectAFinishedJobAndCompare`, at its last step (the click on the review button is not delivered after the project menu closes); row selection and compare pass inside it in the last run and work with real clicks (P, capture), and the review sheet opens with a real click (P, capture) |34| `MacUITests` (9) | 5 passed, 3 skipped, **1 failed** | passed: Settings panes by click, memory create / pin / delete, Research and Lab menus + View menu, notification route, and — since the window-probe fix — saved layout in the menus, re-open, delete and ⌘K. Skipped with a stated reason: both menu-bar-extra tests (the status item is hidden on this Mac's full menu bar), window restoration (done by `Scripts/restore-check.sh`). **Failed**: routing strategy (an XCUITest click on the segmented control is not delivered; with real clicks every click wrote the preference and refreshed the preview in < 0.25 s — P) |3536**Synthesized drags.** XCUITest's `press(…thenDragTo:)` does not start a SwiftUI / AppKit drag session on macOS 27.37The three drag paths were therefore checked with real pointer events (P): a watchlist row dragged below another one38(order changed on screen and on the server, then restored); a trend line drawn by dragging in the plot and a39horizontal line by clicking (capture `build/shots/next/chart-drawn-by-pointer.png`); a ticker dragged from a watchlist40row onto a detached chart window (its title went from AAPL to MSFT).4142**Clicks not delivered by XCUITest** were seen on two custom controls (the kit's segmented control while its thumb43animates, a toolbar button right after a menu closed). Real clicks on the same controls always worked. This is44reported as a test-harness limit, not hidden: the two failing tests stay failing in the suite.4546## Live runs on 5058 (model calls: within the allowance of two fast analyses and one research study)4748| Run | Generation | Model | Usage reported by the server | Seen |49|---|---|---|---|---|50| Fast, `auto` | `fe5fc3c4-7899-499f-8406-93f420f062c1` | routed to GPT-5 Nano (economy strategy at that moment) | 61.5 k in · 826 out · $0.0034 | `model_routed` line with its reason, usage line, provenance, trace |51| Research study, cap 3 USD, with `mac-test-closes.csv` (400 rows) | `b6a72f8d-a9f6-4889-925e-362c4afc9354` | Sonnet 5.5 (7 lead calls) + Haiku 5.5 (1) | 529 815 in · 3 385 out · $0.318 | upload and inspection card, cockpit while running, 4 code cells (one failed, kept with its traceback), 1 experiment, 2 figures with linter verdicts, quality gate, 13 files; reopened later from REST |5253Both were started through the composer's own send path by a DEBUG launch argument, not by a click on Send. The second54fast analysis of the allowance was not used. Not observed in these runs: an approval request, a background run, a55package installation, a budget event, a plan.5657Quant engines (no model cost) were run freely: volatility, risk, regression, options (pricer, strategy, chain),58backtest with walk-forward, portfolio, legacy tools, a validation failure, cancelled jobs.5960## Checked with real pointer and key events (P)6162- Routing strategy clicks in Settings: preference written and router preview refreshed after each of four clicks.63- "Request review" (then named "Faire vérifier") opens the sheet labelled "Automated review — not a human peer review" (nothing was started).64- Drag-reorder in a watchlist; drawing by drag and by click on the chart; ticker drop on a detached chart.65- ⇧⌘N opens a detached chart window with its content.66- Saved window layout, second round (real menu clicks through System Events, name set in the dialog, 5058): before67  the fix the toast read "No window to save."; after it, the same sequence wrote the `workspace` object (one `chart`68  panel, `config.symbol` MSFT, the Mac part in `config.mac`), listed the layout in the Window menu, and choosing it69  closed the window and opened a new one titled "Charting – MSFT" with its content70  (`build/shots/next/layout-reopened.png`). The test object was deleted afterwards.71- Quant Lab, second round: four real clicks on rows of the Computations list changed the selection four times; in72  the two cases where the clicked row could be matched without ambiguity (start time and duration) the highlighted73  row, the job line and the result agree. One earlier click selected another row than the one aimed at: a fixed74  coordinate was used on a list at another scroll position — not accounted for further. The Compare menu listed only75  other results of the same engine, without the current one; choosing one showed the comparison with the right A and B76  (`build/shots/next/lab-compare-by-pointer.png`).77- Window restoration: `Scripts/restore-check.sh` — three main windows on three sections (one opened with ⌘N and moved78  to Statements) and three chart windows; real quit keeping windows; relaunch: the six windows returned with the same79  titles and frames.8081## Seen running (window captures, light and dark)8283`build/shots/next/` (merged tree): overview (light, dark), research, Quant Lab catalogue, chart, chart drawn by84pointer, watchlists, chart window after drop. The engineers' captures of every screen are in85`~/Library/Developer/VQuant/shots/{terminal,research,lab,chart,portfolio-studio,mac}/` (outside the repository):86statements, palette, the six inspector tabs on a recorded and on the live study, approvals sheet (recorded event),87trace sheet, model picker, dataset card, lab forms / running / results of seven engines / compare / options /88backtest / review, portfolios (valuation, import, analysis, rebalancing proposal), studio (project, notes and89versions, reviews, memory, export), the 20 Settings panes, the menu-bar panel in a preview window. Not every capture90of every pair was opened; those opened on the merged tree are the ones listed first.9192## Language check on VQuantKit 0.4 (second round)9394The app builds against VQuantKit 0.4 (`ccade44`) and asks the kit for English. Seen on that build95(`build/shots/next/cockpit-kit04-dark.png`, `lab-result-kit04-light.png`): the kit's own vocabulary is English96(objective, quality gate, timeline, code cells, lab result). A single language was **not** reached at first, for the reasons below; 1 and 2 are corrected, 3 is intended:97981. The six inspector tab titles and the action "Faire vérifier" were French strings of this app (the first brief99   named them so). The app being English-only, they now read Activity · Code · Experiments · Figures · Files ·100   Environment and "Request review" (the sheet keeps its label "Automated review — not a human peer review"). No other101   French literal remains in the app sources (searched for accented literals).1022. The unit of a number of days was printed "j" whatever the language (hard-coded in the kit's formatter). The kit's103   `VQFormat.byUnit` now takes the language (default: the host app's) and reads the suffix from its string catalogue:104   "d" in English, "j" in French; kit tests 163 passed, including the web golden vectors (French) and a new test of105   both suffixes. That kit edit is left uncommitted in the kit repository for the lead to commit.1063. Not a language: numbers follow the account's synced number format (fr-CA on the owner's account). Intended.107108## Defects found and fixed during this work1091101. **A detached chart window opened while signed in was blank** (present since 1.0; 1.0.1's report says detached111   windows were never exercised). Cause: the window's `task` hung on an empty `Group`. Same cause for a main window112   re-opened by a saved layout. Found with real pointer events and the layout UI test.1132. **Restored windows came back on the default section**: scene storage can arrive after the first appearance and114   was only read once. Now applied when it changes, and written for windows that never navigated.1153. ⌘K opened the palette without keyboard focus in its field (UI test).1164. `VQEmptyState` inside a flexible stack pushed the window content off the top (UI test) — empty states are overlays.1175. `/api/quant/jobs` failed to decode as soon as one job was `cancelled` (new status of backend 6) — tolerant decoding.1186. 1.0.1's ⌃⌘S (Save Window Layout) collided with the system's Show/Hide Sidebar → ⇧⌘S; shortcuts now come from one119   table with a unit-tested collision check.1207. `HSplitView` in the lab workbench looped in AppKit constraints when the comparison opened → own divider.1218. An accessibility label on selectable text made macOS 27 recurse under XCUITest whenever a conversation was shown.1229. The watchlist drop indicator stayed after a drop.12310. **A window layout could not always be saved ("No window to save.")**. The helper that hands a window to SwiftUI124    code asked for it once, one run-loop turn after its probe view was created; when the view was not attached yet125    (a main window whose content is inserted after its model exists) the callback never ran: the window was never126    registered for layouts and `WindowModel.window` stayed nil. It now reports from `viewDidMoveToWindow`. Found on the127    second round with real menu clicks; it also explains why the layout UI test passed in some runs and failed in128    others.12911. "1 experiments" in the study header and the inspector summary (counts are now singular or plural).130131## Measurements132133- **Chart renderer** (`docs/CHART_PERFORMANCE.md`, Debug build, M5 Max, 120 Hz, 1000×640 pt, every bar visible, median134  main-thread work per frame): 10 000 bars with all indicators 39.4 ms (1.0.1) → 5.0 ms (2.0); 50 000 bars 164.7 →135  4.8 ms; 100 000 bars 444.3 → 4.4 ms; with 50 drawings 5.6–6.7 ms. 22 of 23 040 frames over the 8.33 ms budget136  across 96 scenarios. Metal was not adopted: the time was in the app's own per-frame work, not in rasterisation.137  Limits: Debug build, other builds running during the measurement, no true 5K display.138- **Memory**: the signed Release build, signed in and showing Research, 8 s after launch: 162 MB resident (one `ps`139  sample; see the note on that launch below). 1.0.1 reported 142 MB idle. No long-session measurement, no Instruments.140- **Launch time**: not measured for 2.0.0 (the one timing taken included the polling tool's own start-up).141- **Bundle**: 24 MB universal (1.0.1: 6.9 MB) — the executable now contains the kit and about 19 000 more lines.142- **Sync**: with real clicks, a preference write reached the server and the dependent server answer was on screen143  60–240 ms after the click (four samples from a temporary log, removed).144145## Release146147- `Scripts/release.sh`: version 2.0.0 (3), universal, signed non-interactively with the existing Developer ID148  Application identity and the "VQuant Developer ID" profile. Bundle id, entitlements, associated domain and signing149  settings are unchanged from 1.0.1 (`Resources/*.entitlements`, `Resources/Info.plist` untouched).150- **Shipped on 2026-11-20 at the owner's request:** notarized by Apple on 2026-11-20 (submissions 7204cfbb… app, 60682e96… disk image, both Accepted), stapled, installed in /Applications and published as `VQuant-2.0.0.dmg` (sha256 044d90ed…) under spbgit releases `vquant-macos/v2.0.0`. Gatekeeper: `accepted, source=Notarized Developer ID`.151  First launch of the installed app (evidence **P**, production backend 6): it opened on the owner's existing session without a152  sign-in prompt and its requests to `/api/v2/{models,market/overview,data/quotes,sync/changes,research/*}` returned 200.153  Nothing else of 2.0.0 has been exercised against production.154155## One unintended launch against production156157To measure launch time and memory, the signed `dist/VQuant.app` was launched once. Having the production bundle id,158it found the owner's existing session in the Keychain and signed in to `vquant.spboucher.ai` for about ten seconds159before being stopped. What it did there: the same read requests a launch of 1.0.1 makes (session, workspace,160conversations, indices, quant tools — now with the header `vquant-macos/2.0.0`) and attempts on `/api/v2` routes that161production (backend 5.8) does not have. Nothing was written to the server: the sync engine never completed a pull162(its replica directory is empty), so no migration and no object write happened. On this Mac it refreshed six files of163the encrypted cache and created the empty directory `Application Support/VQuant/sync-vquant.spboucher.ai/simon` in164the app's container. It was not repeated.165166## Not verified167168- **Touch ID**: every ceremony — sign-in on the signed 2.0.0 build by a person, native enrolment, the app lock,169  revocation and passkey deletion from the Settings panes (the API is tested with a software authenticator).170- **Against production backend 6**: nothing — backend 6 is not in production; 2.0.0 has only met the 5058 copy.171- **Research**: an approval answered from the Mac and its notification (only a recorded event was rendered);172  attaching to a study running on another device; background runs, their logs and cancel; budget events;173  `enforced: false` banner; the 413 sentence; Quick Look, drag-out to the Finder, Save As in the file browser;174  deep and ultra modes, Stop and mid-stream join on backend 6; exports, share, image attachment (code carried over175  from 1.0.1, not re-exercised).176- **Quant Lab**: starting a review; add to project; exports and the save panel; "ask the agent"; the limit-of-3177  message; the background notification.178- **Studio / Portfolios**: a research run filed in a project; drag-reorder of items; "Apply to a Copy"; the file179  importer and CSV drop; save panels; the `main` portfolio flows (it is empty on the test copy) and the Risk180  Analytics refresh after a sync write to `main`.181- **Terminal**: quarterly / balance / cash-flow statement tabs in a capture; CSV export and copy; every context182  menu; palette routes to notes / portfolios / charts / projects and the recents write.183- **Sync**: the migration on a Mac that really holds 1.x watchlists or layouts (the rule is unit-tested against the184  kit's fake server; the app's own function was never run on such data); a real conflict notice; offline edits with185  the network physically off; two real Macs.186- **Mac**: the real menu-bar item (hidden on this Mac) and its polling cadence; a system notification delivered and187  clicked (only the router behind it); saved layouts from the menus beyond the UI test that passed once.188- Offline reading with the network physically off; VoiceOver; Intel Macs; macOS 14–26; a second person's account;189  long sessions; Instruments.190191## Left on the test backend (5058)192193Test objects kept so the screens can be reproduced: portfolios `mac-test-pf-multi`, `mac-test-pf-usd`; projects194`mac-test-studio`, `mac-test-lab-project`; notes `mac-test-note-1`, `mac-test-note-2`; two project memories; research195workspaces named `mac-test-…` and one "Untitled study" (no delete route); a few dozen quant jobs; the two generations196above. `preference/default` did not exist before this work: it now holds the routing `balanced` with the three197default providers.198
Files main
Menu

Theme