1# FinData OS — Discovery report (Phase 0)23Date: 2026-11-01 (America/Toronto). Author: Claude Code session for the owner (spbou4@icloud.com).4All vendor probes below were run **from the storage node M3U96a**, with the vendor user id read from5`~/findata-secrets/frd.env` (mode 600) and redacted as `<KEY>` in every log and document.67## 1. Cluster inventory and storage node selection89Measured on 2026-11-01 (`df -H /`, `diskutil info disk0`, `uptime`, `pm2 jlist`). Three nodes were unreachable at10probe time (M4M36, M2M32, M2M32b — SSH timeout) and were excluded.1112| Node | SSD | Free | RAM | Cores | SMART | Load (1m) | Already hosts |13|---|---|---:|---:|---:|---|---:|---|14| **M3U96a** | APPLE SSD AP1024Z (1 TB) | **508 GB** | 96 GB | 32 | Verified | 1.8 | admin-ka, vrai-prix, job-ka, sorti-ka, qc26 (≈2 GB RSS total) |15| M3U96b | APPLE SSD AP1024Z (1 TB) | 424 GB | 96 GB | 32 | Verified | 1.8 | toveli, autoscholar, UQO_RESEARCH_DB 232 GB, ovh-retired backups 26 GB |16| M4M64a | APPLE SSD AP1024Z (1 TB) | 778 GB | 64 GB | 16 | Verified | 2.2 | lou-ka, fabri-ka, ka-stats, uqo-chat, pdb-api, maclustr-www, agentd, maclustr-mobile |17| M2U64 | APPLE SSD AP1024Z (1 TB) | 339 GB | 64 GB | 24 | Verified | 1.6 | spb-cloud, vquant, uqo-eval, console, appmanager, nodepages, spboucher-ai, food-ka |18| M4M64c | APPLE SSD AP0512Z (512 GB) | 342 GB | 64 GB | 16 | Verified | 2.2 | akos, chat, note |19| others | 512 GB class | 216–448 GB | 16–48 GB | 10–18 | Verified | — | — |2021**Decision: M3U96a.** Rationale:22- Largest compute (32 cores, 96 GB RAM) for DuckDB/Polars analytical reads, with 1 TB internal NVMe (SMART verified).23- Measured sequential write 5.1 GB/s (`dd` 2 GiB), read 21.8 GB/s (page-cache assisted) — more than enough for Parquet compaction.24- 508 GB free fits the capacity model below with ≥2× headroom; M4M64a has more free space but is the agentd/mobile25 coordinator node, carries the heaviest Ka apps (lou-ka 1.3 GB RSS) and has half the cores.26- PostgreSQL 17.10, Redis 8.8, uv, pm2 6 and Node 25 are already installed; wg1 tunnel peer 10.67.0.10 is up (handshake < 2 min).27- Backup target: **M4M64a** (778 GB free, same LAN, 1 TB SSD) via rsync; restore drill documented in `docs/OPERATIONS.md`.28- Nothing is wiped or repartitioned: the lake lives in `~/findata-lake/` on the existing APFS volume.2930Network: LAN 192.168.2.x (1 Gbit+), public egress via Bell Giga Hub (TCP 80/443 + UDP 53 only; WireGuard to BHS64 uses UDP 443).31Reverse proxy / TLS: Caddy on BHS64 (OVH Beauharnois), WireGuard hub 10.67.0.1. DNS `findata.spboucher.ai → 51.161.112.61` **verified32present** (`dig +short`) before any change; the route is added with `mld deploy` (`tunnel.domain` in the manifest), no other route touched.33Existing auth/monitoring: none shared across apps; agentd (M4M64a:9210) polls PM2 + health paths; `mld heal` every 5 min.3435## 2. Vendor identification3637"First Trade Data" = **FirstRate Data** (firstratedata.com). Evidence: the doc URL given by the owner38(`https://firstratedata.com/c/market-data-api/<KEY>`), the prior platform `hfmarketdata` (spbgit `hfmarketdata.git`, retired 2026-10-02)39whose `frd_downloader.py` used `FRD_USERID` against `https://firstratedata.com/api`, and the official docs at40`https://firstratedata.com/about/api-docs`, `/about/license`, `/about/FAQ`, `/_readme/stock.txt`, `/fundamental-data/*`.4142Auth: the user id is a **query parameter** (`userid=<KEY>`) on both the market-data API and the fundamentals API (`/api-f`, where the43page says `api_key=` but only `userid=` is accepted — `api_key=` returned HTTP 403 on 2026-11-01). The key therefore appears in URLs:44the connector never logs full URLs, redacts the parameter in every exception message, and the HTTP client disables request logging.4546## 3. Capability matrix (verified 2026-11-01)4748Legend: **confirmed** = probed with the owner's key and a real response; **doc** = documented, not yet exercised; **unsupported** = vendor49says no; **unknown** = not determinable without a paid/bulk call.5051| Capability | Status | Evidence |52|---|---|---|53| `last_update?type=stock|etf|futures|index|options` | confirmed | all return `2026-10-30` (crypto `2026-10-31`, fx returns `20261030` — different format!) |54| `ticker_listing?type=stock` | confirmed | 16 313 rows (`Ticker,Name,First Date,Last Date`), 8 069 `-DELISTED` suffixes |55| `ticker_listing?type=etf` | confirmed | 5 100 rows |56| `update_log?type=stock&log_type=delisted` | confirmed | 1 355 rows (`Ticker,Last Date`) |57| `meta_file splits` / `dividends` | confirmed | 302 → zip (Backblaze B2) of per-ticker `{date,ratio}` / `{exdate,amount}` files: 2 963 / 3 900 files |58| `data_file type=stock period=day timeframe=1min adjustment=UNADJUSTED` | confirmed | 302 → `https://x01.frd-data.com/…zip`, 15.76 MB zip, 6 515 files, 66.4 MB raw, no header, `yyyy-MM-dd HH:mm:ss,O,H,L,C,V`, 04:00–19:59 ET (extended hours), bar **open** timestamps |59| `data_file period=full&ticker_range=A-Z` (stock/etf, 1min/5min/30min/1hour/1day, adj_split/adj_splitdiv/UNADJUSTED*) | doc | *UNADJUSTED only 1min + 1day. Bundle size ≈ 34 GB zip / 380 GB raw for stocks+ETFs. **Not downloaded yet** (bounded backfill in Phase 5 only) |60| `full_update period=day|week` (re-adjusted histories) | doc | needed to repair adjusted series after splits/dividends |61| `delisted_data_file archive_number=1..5 / update=week|year` | doc | archives count inconsistent in docs (4 vs 5) |62| futures (continuous ×3 adjustments + individual contracts), index, fx, crypto | confirmed license (last_update) ; doc endpoints | FX intraday = `yyyyMMdd,HH:mm:ss,…` two-field datetime; crypto timestamps UTC; index has no volume; futures daily bars follow the contract trading day |63| options (EOD only, CBOE feed, Greeks) | confirmed license ; doc | no intraday options; bundle only; 70 GB zip incl. options |64| quotes / trades / ticks | **unsupported** | vendor provides bars only |65| Fundamentals `/api-f/v1/financials/all/{ticker}/{annual|quarterly}` | confirmed | AAPL: 14 annual / 59 quarterly records, 76 fields incl. `period_end`, `filing_date`, `fiscal_year`, `fiscal_period`, `actual_year`, `actual_period` |66| `/api-f/v1/financials/single/{ticker}?period=FY|Q1..Q4&year=` | doc | |67| `/api-f/v1/earnings_release/t/{ticker}`, `/calendar/{start}/{end}` | confirmed | AAPL 115 records (`date, eps_estimate, eps_reported, release_time`) |68| `/api-f/v1/metrics/t/{ticker}/{start}/{end}`, `/metrics/last` | confirmed | daily valuation metrics (pe_ttm, market_cap, ev, …) |69| `/api-f/v1/short_interest`, `/short_volume`, `/social_sentiment` | doc | parameter tables only |70| Fundamentals revisions / restatement history | **unknown** | vendor exposes one value per (ticker, period) with `filing_date`; no "as originally reported vs restated" flag → we snapshot each ingestion run and version rows ourselves (see ARCHITECTURE §PIT) |71| Shares outstanding | confirmed (field `shares`, `shares_av`, `shares_av_diluted`) | |72| Corporate actions | confirmed splits + cash dividends only | no mergers/spin-offs |73| Identifier history | partial | ticker changes merged under latest ticker (since 2022), delisted suffixed `-DELISTED`; no CUSIP/ISIN/CIK from vendor → CIK mapping via SEC `company_tickers.json` (connector `sec_edgar`, pending configuration) |74| Rate limits / quotas | none documented ("There are no rate limits") | we still budget 2 concurrent downloads, 1 req/s metadata |75| Pagination | none | whole zip archives |76| Historical depth | stocks from 2000 (ticker listing first dates), options 2010, short data 2021 | |77| Update schedule | docs conflict (23:30 ET / 01:00 ET / 02:00–03:00 ET for full) | policy: poll `last_update` before every incremental job |7879Sanitized probe transcripts: `docs/evidence/phase0-probes.md`.8081## 4. License (FirstRate Data, `/about/license`, `/about/FAQ`)8283- Perpetual private use; data may be stored on own servers, loaded into databases, used for research, backtests, model training, derived84 statistics and charts. Attribution to FirstRate Data required when publishing research.85- **Prohibited**: redistributing raw data to third parties (CSV downloads, an API returning bars, tables of prices/volumes), reselling, sharing86 the user id. Multi-user only within a single institution with a multi-user license.87- Consequence for FinData OS: **owner-only by default**; every API token is scoped; the console shows a license banner; no public routes88 return bars; exports are owner-scoped and expire. The `license_class` column is `frd-private` on every vendor row.8990## 5. Capacity model (1-minute data)9192Measured: one full trading day of 1-min UNADJUSTED bars for 6 515 stocks = 66.4 MB CSV (15.8 MB zip). Parquet ZSTD(3) with typed93columns compresses FirstRate CSV ≈ 5–6× (prior hfmarketdata lake: 26.5 G rows in 350 GB incl. options → ≈ 13 bytes/row).9495| Scenario | Rows | Raw zip kept | Curated Parquet | Total incl. 15 % index/manifests + backup copy ×1 |96|---|---:|---:|---:|---:|97| Stocks, 1-min UNADJUSTED, 1 year incremental (252 d) | ≈ 7.5 × 10⁷ × … ≈ 1.6 × 10⁸ (avg 100 bars/sym/day × 6 500 × 252) | 4 GB | 2.5 GB | ≈ 15 GB |98| Stocks + ETFs, 1-min, full history 2000→2026, one adjustment | ≈ 1.5 × 10¹⁰ | 34 GB (bundle) | ≈ 190 GB | ≈ 500 GB (with backup on M4M64a) |99| + 5-min derived | +20 % | — | +40 GB | |100| Full vendor bundle all timeframes/adjustments (not planned) | ≈ 2.6 × 10¹⁰ | 70 GB | 350 GB | > 800 GB — exceeds single-node budget |101102Budget rule adopted: lake ≤ 300 GB on M3U96a (alert at 250 GB, hard stop at 60 GB free), raw zips retained for **incremental** archives103only (immutability proof) and deleted after checksum + Parquet publication for **full** archives (the zip is re-downloadable; its SHA-256 is104kept in the manifest). Full backfill is executed by letter range and by adjustment on explicit operator request (`findata ingest backfill`).105106## 6. Existing code and integration points107108- `hfmarketdata.git` (spbgit): `frd_downloader.py` (job plan, resume manifest, CSV→Parquet via DuckDB, zip quirks: nested zips, mixed109 LF/CRLF, FX two-field datetime, volume decimals) and `hfmarketdata/api/fundamentals/edgar_client.py` (SEC companyfacts) are reused as110 knowledge; code is rewritten under the FinData contracts (raw immutability, provenance, PIT).111- `mld` (M1M32): deployment, tunnel route, heal. Manifest `M1M32:~/dispatch/apps/findata.json`.112- agentd (M4M64a): auto-discovers PM2 processes and health paths from the registry.113- Postgres 17 local on M3U96a (`findata` role + database created 2026-11-01, localhost only). Redis 8 available (not used in v1: job queue114 is Postgres-backed with `SELECT … FOR UPDATE SKIP LOCKED`, durable across reboots).115116## 7. Blocked / deferred (honest list)117118- Fundamentals **revision history**: not provided by the vendor; FinData versions rows per ingestion run (`observed_at`) and flags value119 changes as `restatement_flag=true`. Historical restatements before the first ingestion are unknowable from this vendor.120- SEC EDGAR connector: implemented as a modular connector with fixture tests, **pending configuration** (requires a declared User-Agent121 contact e-mail per SEC fair-access policy; no secret).122- Full multi-year 1-min backfill: deferred to Phase 5 (representative letter range), on operator request because of disk budget.123- Options, futures contracts, FX, crypto, index: capability confirmed; connectors implement the file layouts; **ingestion not run** in124 Phase 2 (one licensed dataset end-to-end = US stocks 1-min UNADJUSTED).125