PHASE 0 — Audit report (2026-11-02)
Scope
Inspect the repository, the live FinData OS API, the dataset catalog, the data license and the cluster resources; publish an exact feasibility report with risks. No data were ingested in this phase beyond small verification pages.
Repository
Recherche-UQO/holds 40 working papers (README index);finance/has 14 projects including WP39 (FinData, 1-min vs daily for accounting deterioration) and WP40 (RAID, C++/Metal econometrics). Nothing was modified outside the new directoryfinance/wp41_uqo/(own git repository; remotewp41_uqo.gitto be created on spbgit at release).- Template conventions (
HARMONISATION.md): title page macros,paper/Makefile+.latexmkrc, README skeleton; copied.
Live API (https://findata.spboucher.ai/api/v1, probed 2026-11-02 22:26 UTC; results/metrics/p0_api_audit.json)
| Item | Verified value |
|---|---|
| OpenAPI | 3.1.0, 41 paths; ExportIn = {dataset_id, symbols ≤ 500, start, end, adjustment ∈ {raw, split, split_div}, format} |
| Bars schema | v1, timestamp_convention = bar_open, storage UTC, 20 columns incl. quality_flags, revision_id |
| Health | status ok, no alerts; lake 564.65 GB curated, 259.7 GB free (soft limit 1 200 GB) |
market_bars.stock.1m.raw |
active, 6 695 639 332 rows, 2000-01-03 → 2026-10-30, license class frd-private |
market_bars.etf.1m.raw |
active, 907 699 950 rows, 2000-01-03 → 2026-10-30 |
| Pagination | 1m ≤ 31 days per GET /market/bars call, ≤ 10 000 rows/page, x-next-cursor; Arrow IPC transport |
| Rate limit | 600 req/min per key (client token bucket 5–8 req/s) |
| Headers preserved | X-Request-ID, X-Dataset-Id, X-Dataset-Version (17 375 for stock 1m at audit time) |
Timestamp conventions on real rows (AAPL, SPY):
| Session | Regular rows | Expected | First UTC bar | Closing print volume vs median |
|---|---|---|---|---|
| 2024-03-08 (EST) | 391 | 391 | 14:30Z | 6.72 M vs 0.12 M |
| 2024-03-11 (EDT) | 391 | 391 | 13:30Z | 6.61 M vs 0.07 M |
| 2024-11-29 (13:00 early close) | 211 | 211 | 14:30Z | 6.33 M vs 0.05 M |
| SPY 2024-07-03 (early close) | 211 | 211 | 13:30Z | 0.89 M vs 0.06 M |
Split check: raw/split close ratio on 2020-08-28 (AAPL 4:1 on 2020-08-31) = 4.000, volume ratio 4.000 → explicit
split adjustment reproduces the server-side series (live test test_explicit_split_equals_server_split).
Extended hours: bars from 04:00 ET (08:00Z/09:00Z) to 19:59 ET are present and excluded by the regular grid.
All probed rows have empty quality_flags and revision_id = 1.
License (FirstRate Data, DISCOVERY.md §4)
Private use, research and model training allowed; no redistribution of raw bars (no CSV, no API returning bars,
no tables of prices/volumes). Consequence: the release contains code, aggregate metrics, figures of scores and
synthetic fixtures only; caches live outside git on the compute node (docs/LICENSE.md).
Finnhub
Not used. No credentials were requested or verified; the brief treats it as an optional, separate vendor. Decision:
out of scope for v1.0 (documented in docs/DATA_CARD.md).
Cluster resources (probed 2026-11-02 17:27 ET)
| Node | Chip | CPU/GPU | RAM | Free disk | Load | Role chosen |
|---|---|---|---|---|---|---|
| M3U96a | Apple M3 Ultra | 28 c / 60 GPU | 96 GB | 260 GB | 1.4 | FinData host (API :8600); not used for training |
| M3U96b | Apple M3 Ultra | 28 c (20P+8E) / 60 GPU | 96 GB | 423 GB | 1.8 | compute node: ~/wp41_uqo, venv Python 3.12, torch 2.15.0 (MPS available) |
| laptop | Apple M5 Max | 40 GPU | — | — | — | authoring, tests, LaTeX |
LAN path M3U96b → M3U96a:8600 health = 9 ms. API key: dedicated wp41_uqo key (scopes read:market, read:fundamentals, |
||||||
query, export) created with findata create-key on the FinData node, stored mode 600 at M3U96b:~/findata-secrets/api-key-wp41.txt |
||||||
| and in the laptop secret store; never in git. |
Capacity and feasibility estimate
- Upper bound in the brief: 300 × 252 × 390 × 8 ≈ 236 M minute rows. Export throughput measured on the pilot (20 symbols × 84 sessions = 473 923 rows): 29.6 s wall with 3 workers → ≈ 16 k rows/s/worker including polling; main sample ≈ 236 M rows ≈ 1–2 h of exports; Parquet ≈ 13 B/row → ≈ 3 GB; dense grids (float32 base + masks) ≈ 16 MB per symbol-decade → ≈ 5 GB. Disk budget 423 GB: feasible with > 50× headroom.
- Universe (P1,
data/manifests/universe_main.json): 2 147 candidates passing the PIT rules, 300 selected, 60 held out; only 12 delisted candidates exist (delisted 1-minute archives are only partially ingested in FinData) → survivorship is a documented limitation, not a solved problem. - Pilot grids: regular-hours minute coverage ranged from 27 % (HEZU) to 100 % (WMT, XLU) → a coverage audit over the full window is running for the whole candidate pool before the universe is frozen (P1 gate).
Risks
- Sparse minute coverage for low-activity instruments (masks handle it, but origin eligibility shrinks).
- Survivorship: the vendor's delisted archives are not fully in the lake.
- MPS numerical behaviour (no FP64; Cholesky on CPU) — likelihoods kept in FP32, copula linear algebra on CPU.
- Compute: a single node; the brief's 15–30 M model on 236 M rows must be budgeted from measured throughput (P4).
- No quotes: economic evaluation is assumption-based only.
Go / no-go
GO to P1 with the connector contract above. Tests at the end of P0: 16 offline tests + 5 live contract tests pass
(pytest, WP41_LIVE=1 pytest -m live).