FinData OS — Discovery report (Phase 0)
Date: 2026-11-01 (America/Toronto). Author: Claude Code session for the owner (spbou4@icloud.com).
All vendor probes below were run from the storage node M3U96a, with the vendor user id read from
~/findata-secrets/frd.env (mode 600) and redacted as <KEY> in every log and document.
1. Cluster inventory and storage node selection
Measured on 2026-11-01 (df -H /, diskutil info disk0, uptime, pm2 jlist). Three nodes were unreachable at
probe time (M4M36, M2M32, M2M32b — SSH timeout) and were excluded.
| Node | SSD | Free | RAM | Cores | SMART | Load (1m) | Already hosts |
|---|---|---|---|---|---|---|---|
| M3U96a | APPLE SSD AP1024Z (1 TB) | 508 GB | 96 GB | 32 | Verified | 1.8 | admin-ka, vrai-prix, job-ka, sorti-ka, qc26 (≈2 GB RSS total) |
| M3U96b | APPLE SSD AP1024Z (1 TB) | 424 GB | 96 GB | 32 | Verified | 1.8 | toveli, autoscholar, UQO_RESEARCH_DB 232 GB, ovh-retired backups 26 GB |
| M4M64a | APPLE SSD AP1024Z (1 TB) | 778 GB | 64 GB | 16 | Verified | 2.2 | lou-ka, fabri-ka, ka-stats, uqo-chat, pdb-api, maclustr-www, agentd, maclustr-mobile |
| M2U64 | APPLE SSD AP1024Z (1 TB) | 339 GB | 64 GB | 24 | Verified | 1.6 | spb-cloud, vquant, uqo-eval, console, appmanager, nodepages, spboucher-ai, food-ka |
| M4M64c | APPLE SSD AP0512Z (512 GB) | 342 GB | 64 GB | 16 | Verified | 2.2 | akos, chat, note |
| others | 512 GB class | 216–448 GB | 16–48 GB | 10–18 | Verified | — | — |
Decision: M3U96a. Rationale:
- Largest compute (32 cores, 96 GB RAM) for DuckDB/Polars analytical reads, with 1 TB internal NVMe (SMART verified).
- Measured sequential write 5.1 GB/s (
dd2 GiB), read 21.8 GB/s (page-cache assisted) — more than enough for Parquet compaction. - 508 GB free fits the capacity model below with ≥2× headroom; M4M64a has more free space but is the agentd/mobile coordinator node, carries the heaviest Ka apps (lou-ka 1.3 GB RSS) and has half the cores.
- PostgreSQL 17.10, Redis 8.8, uv, pm2 6 and Node 25 are already installed; wg1 tunnel peer 10.67.0.10 is up (handshake < 2 min).
- Backup target: M4M64a (778 GB free, same LAN, 1 TB SSD) via rsync; restore drill documented in
docs/OPERATIONS.md. - Nothing is wiped or repartitioned: the lake lives in
~/findata-lake/on the existing APFS volume.
Network: LAN 192.168.2.x (1 Gbit+), public egress via Bell Giga Hub (TCP 80/443 + UDP 53 only; WireGuard to BHS64 uses UDP 443).
Reverse proxy / TLS: Caddy on BHS64 (OVH Beauharnois), WireGuard hub 10.67.0.1. DNS findata.spboucher.ai → 51.161.112.61 verified
present (dig +short) before any change; the route is added with mld deploy (tunnel.domain in the manifest), no other route touched.
Existing auth/monitoring: none shared across apps; agentd (M4M64a:9210) polls PM2 + health paths; mld heal every 5 min.
2. Vendor identification
"First Trade Data" = FirstRate Data (firstratedata.com). Evidence: the doc URL given by the owner
(https://firstratedata.com/c/market-data-api/<KEY>), the prior platform hfmarketdata (spbgit hfmarketdata.git, retired 2026-10-02)
whose frd_downloader.py used FRD_USERID against https://firstratedata.com/api, and the official docs at
https://firstratedata.com/about/api-docs, /about/license, /about/FAQ, /_readme/stock.txt, /fundamental-data/*.
Auth: the user id is a query parameter (userid=<KEY>) on both the market-data API and the fundamentals API (/api-f, where the
page says api_key= but only userid= is accepted — api_key= returned HTTP 403 on 2026-11-01). The key therefore appears in URLs:
the connector never logs full URLs, redacts the parameter in every exception message, and the HTTP client disables request logging.
3. Capability matrix (verified 2026-11-01)
Legend: confirmed = probed with the owner's key and a real response; doc = documented, not yet exercised; unsupported = vendor says no; unknown = not determinable without a paid/bulk call.
| Capability | Status | Evidence |
|---|---|---|
| `last_update?type=stock | etf | futures |
ticker_listing?type=stock |
confirmed | 16 313 rows (Ticker,Name,First Date,Last Date), 8 069 -DELISTED suffixes |
ticker_listing?type=etf |
confirmed | 5 100 rows |
update_log?type=stock&log_type=delisted |
confirmed | 1 355 rows (Ticker,Last Date) |
meta_file splits / dividends |
confirmed | 302 → zip (Backblaze B2) of per-ticker {date,ratio} / {exdate,amount} files: 2 963 / 3 900 files |
data_file type=stock period=day timeframe=1min adjustment=UNADJUSTED |
confirmed | 302 → https://x01.frd-data.com/…zip, 15.76 MB zip, 6 515 files, 66.4 MB raw, no header, yyyy-MM-dd HH:mm:ss,O,H,L,C,V, 04:00–19:59 ET (extended hours), bar open timestamps |
data_file period=full&ticker_range=A-Z (stock/etf, 1min/5min/30min/1hour/1day, adj_split/adj_splitdiv/UNADJUSTED*) |
doc | *UNADJUSTED only 1min + 1day. Bundle size ≈ 34 GB zip / 380 GB raw for stocks+ETFs. Not downloaded yet (bounded backfill in Phase 5 only) |
| `full_update period=day | week` (re-adjusted histories) | doc |
| `delisted_data_file archive_number=1..5 / update=week | year` | doc |
| futures (continuous ×3 adjustments + individual contracts), index, fx, crypto | confirmed license (last_update) ; doc endpoints | FX intraday = yyyyMMdd,HH:mm:ss,… two-field datetime; crypto timestamps UTC; index has no volume; futures daily bars follow the contract trading day |
| options (EOD only, CBOE feed, Greeks) | confirmed license ; doc | no intraday options; bundle only; 70 GB zip incl. options |
| quotes / trades / ticks | unsupported | vendor provides bars only |
| Fundamentals `/api-f/v1/financials/all/{ticker}/{annual | quarterly}` | confirmed |
| `/api-f/v1/financials/single/{ticker}?period=FY | Q1..Q4&year=` | doc |
/api-f/v1/earnings_release/t/{ticker}, /calendar/{start}/{end} |
confirmed | AAPL 115 records (date, eps_estimate, eps_reported, release_time) |
/api-f/v1/metrics/t/{ticker}/{start}/{end}, /metrics/last |
confirmed | daily valuation metrics (pe_ttm, market_cap, ev, …) |
/api-f/v1/short_interest, /short_volume, /social_sentiment |
doc | parameter tables only |
| Fundamentals revisions / restatement history | unknown | vendor exposes one value per (ticker, period) with filing_date; no "as originally reported vs restated" flag → we snapshot each ingestion run and version rows ourselves (see ARCHITECTURE §PIT) |
| Shares outstanding | confirmed (field shares, shares_av, shares_av_diluted) |
|
| Corporate actions | confirmed splits + cash dividends only | no mergers/spin-offs |
| Identifier history | partial | ticker changes merged under latest ticker (since 2022), delisted suffixed -DELISTED; no CUSIP/ISIN/CIK from vendor → CIK mapping via SEC company_tickers.json (connector sec_edgar, pending configuration) |
| Rate limits / quotas | none documented ("There are no rate limits") | we still budget 2 concurrent downloads, 1 req/s metadata |
| Pagination | none | whole zip archives |
| Historical depth | stocks from 2000 (ticker listing first dates), options 2010, short data 2021 | |
| Update schedule | docs conflict (23:30 ET / 01:00 ET / 02:00–03:00 ET for full) | policy: poll last_update before every incremental job |
Sanitized probe transcripts: docs/evidence/phase0-probes.md.
4. License (FirstRate Data, /about/license, /about/FAQ)
- Perpetual private use; data may be stored on own servers, loaded into databases, used for research, backtests, model training, derived statistics and charts. Attribution to FirstRate Data required when publishing research.
- Prohibited: redistributing raw data to third parties (CSV downloads, an API returning bars, tables of prices/volumes), reselling, sharing the user id. Multi-user only within a single institution with a multi-user license.
- Consequence for FinData OS: owner-only by default; every API token is scoped; the console shows a license banner; no public routes
return bars; exports are owner-scoped and expire. The
license_classcolumn isfrd-privateon every vendor row.
5. Capacity model (1-minute data)
Measured: one full trading day of 1-min UNADJUSTED bars for 6 515 stocks = 66.4 MB CSV (15.8 MB zip). Parquet ZSTD(3) with typed columns compresses FirstRate CSV ≈ 5–6× (prior hfmarketdata lake: 26.5 G rows in 350 GB incl. options → ≈ 13 bytes/row).
| Scenario | Rows | Raw zip kept | Curated Parquet | Total incl. 15 % index/manifests + backup copy ×1 |
|---|---|---|---|---|
| Stocks, 1-min UNADJUSTED, 1 year incremental (252 d) | ≈ 7.5 × 10⁷ × … ≈ 1.6 × 10⁸ (avg 100 bars/sym/day × 6 500 × 252) | 4 GB | 2.5 GB | ≈ 15 GB |
| Stocks + ETFs, 1-min, full history 2000→2026, one adjustment | ≈ 1.5 × 10¹⁰ | 34 GB (bundle) | ≈ 190 GB | ≈ 500 GB (with backup on M4M64a) |
| + 5-min derived | +20 % | — | +40 GB | |
| Full vendor bundle all timeframes/adjustments (not planned) | ≈ 2.6 × 10¹⁰ | 70 GB | 350 GB | > 800 GB — exceeds single-node budget |
Budget rule adopted: lake ≤ 300 GB on M3U96a (alert at 250 GB, hard stop at 60 GB free), raw zips retained for incremental archives
only (immutability proof) and deleted after checksum + Parquet publication for full archives (the zip is re-downloadable; its SHA-256 is
kept in the manifest). Full backfill is executed by letter range and by adjustment on explicit operator request (findata ingest backfill).
6. Existing code and integration points
hfmarketdata.git(spbgit):frd_downloader.py(job plan, resume manifest, CSV→Parquet via DuckDB, zip quirks: nested zips, mixed LF/CRLF, FX two-field datetime, volume decimals) andhfmarketdata/api/fundamentals/edgar_client.py(SEC companyfacts) are reused as knowledge; code is rewritten under the FinData contracts (raw immutability, provenance, PIT).mld(M1M32): deployment, tunnel route, heal. ManifestM1M32:~/dispatch/apps/findata.json.- agentd (M4M64a): auto-discovers PM2 processes and health paths from the registry.
- Postgres 17 local on M3U96a (
findatarole + database created 2026-11-01, localhost only). Redis 8 available (not used in v1: job queue is Postgres-backed withSELECT … FOR UPDATE SKIP LOCKED, durable across reboots).
7. Blocked / deferred (honest list)
- Fundamentals revision history: not provided by the vendor; FinData versions rows per ingestion run (
observed_at) and flags value changes asrestatement_flag=true. Historical restatements before the first ingestion are unknowable from this vendor. - SEC EDGAR connector: implemented as a modular connector with fixture tests, pending configuration (requires a declared User-Agent contact e-mail per SEC fair-access policy; no secret).
- Full multi-year 1-min backfill: deferred to Phase 5 (representative letter range), on operator request because of disk budget.
- Options, futures contracts, FX, crypto, index: capability confirmed; connectors implement the file layouts; ingestion not run in Phase 2 (one licensed dataset end-to-end = US stocks 1-min UNADJUSTED).