# FinData OS — Discovery report (Phase 0)

Date: 2026-11-01 (America/Toronto). Author: Claude Code session for the owner (spbou4@icloud.com).
All vendor probes below were run **from the storage node M3U96a**, with the vendor user id read from
`~/findata-secrets/frd.env` (mode 600) and redacted as `<KEY>` in every log and document.

## 1. Cluster inventory and storage node selection

Measured on 2026-11-01 (`df -H /`, `diskutil info disk0`, `uptime`, `pm2 jlist`). Three nodes were unreachable at
probe time (M4M36, M2M32, M2M32b — SSH timeout) and were excluded.

| Node | SSD | Free | RAM | Cores | SMART | Load (1m) | Already hosts |
|---|---|---:|---:|---:|---|---:|---|
| **M3U96a** | APPLE SSD AP1024Z (1 TB) | **508 GB** | 96 GB | 32 | Verified | 1.8 | admin-ka, vrai-prix, job-ka, sorti-ka, qc26 (≈2 GB RSS total) |
| M3U96b | APPLE SSD AP1024Z (1 TB) | 424 GB | 96 GB | 32 | Verified | 1.8 | toveli, autoscholar, UQO_RESEARCH_DB 232 GB, ovh-retired backups 26 GB |
| M4M64a | APPLE SSD AP1024Z (1 TB) | 778 GB | 64 GB | 16 | Verified | 2.2 | lou-ka, fabri-ka, ka-stats, uqo-chat, pdb-api, maclustr-www, agentd, maclustr-mobile |
| M2U64 | APPLE SSD AP1024Z (1 TB) | 339 GB | 64 GB | 24 | Verified | 1.6 | spb-cloud, vquant, uqo-eval, console, appmanager, nodepages, spboucher-ai, food-ka |
| M4M64c | APPLE SSD AP0512Z (512 GB) | 342 GB | 64 GB | 16 | Verified | 2.2 | akos, chat, note |
| others | 512 GB class | 216–448 GB | 16–48 GB | 10–18 | Verified | — | — |

**Decision: M3U96a.** Rationale:
- Largest compute (32 cores, 96 GB RAM) for DuckDB/Polars analytical reads, with 1 TB internal NVMe (SMART verified).
- Measured sequential write 5.1 GB/s (`dd` 2 GiB), read 21.8 GB/s (page-cache assisted) — more than enough for Parquet compaction.
- 508 GB free fits the capacity model below with ≥2× headroom; M4M64a has more free space but is the agentd/mobile
  coordinator node, carries the heaviest Ka apps (lou-ka 1.3 GB RSS) and has half the cores.
- PostgreSQL 17.10, Redis 8.8, uv, pm2 6 and Node 25 are already installed; wg1 tunnel peer 10.67.0.10 is up (handshake < 2 min).
- Backup target: **M4M64a** (778 GB free, same LAN, 1 TB SSD) via rsync; restore drill documented in `docs/OPERATIONS.md`.
- Nothing is wiped or repartitioned: the lake lives in `~/findata-lake/` on the existing APFS volume.

Network: LAN 192.168.2.x (1 Gbit+), public egress via Bell Giga Hub (TCP 80/443 + UDP 53 only; WireGuard to BHS64 uses UDP 443).
Reverse proxy / TLS: Caddy on BHS64 (OVH Beauharnois), WireGuard hub 10.67.0.1. DNS `findata.spboucher.ai → 51.161.112.61` **verified
present** (`dig +short`) before any change; the route is added with `mld deploy` (`tunnel.domain` in the manifest), no other route touched.
Existing auth/monitoring: none shared across apps; agentd (M4M64a:9210) polls PM2 + health paths; `mld heal` every 5 min.

## 2. Vendor identification

"First Trade Data" = **FirstRate Data** (firstratedata.com). Evidence: the doc URL given by the owner
(`https://firstratedata.com/c/market-data-api/<KEY>`), the prior platform `hfmarketdata` (spbgit `hfmarketdata.git`, retired 2026-10-02)
whose `frd_downloader.py` used `FRD_USERID` against `https://firstratedata.com/api`, and the official docs at
`https://firstratedata.com/about/api-docs`, `/about/license`, `/about/FAQ`, `/_readme/stock.txt`, `/fundamental-data/*`.

Auth: the user id is a **query parameter** (`userid=<KEY>`) on both the market-data API and the fundamentals API (`/api-f`, where the
page says `api_key=` but only `userid=` is accepted — `api_key=` returned HTTP 403 on 2026-11-01). The key therefore appears in URLs:
the connector never logs full URLs, redacts the parameter in every exception message, and the HTTP client disables request logging.

## 3. Capability matrix (verified 2026-11-01)

Legend: **confirmed** = probed with the owner's key and a real response; **doc** = documented, not yet exercised; **unsupported** = vendor
says no; **unknown** = not determinable without a paid/bulk call.

| Capability | Status | Evidence |
|---|---|---|
| `last_update?type=stock|etf|futures|index|options` | confirmed | all return `2026-10-30` (crypto `2026-10-31`, fx returns `20261030` — different format!) |
| `ticker_listing?type=stock` | confirmed | 16 313 rows (`Ticker,Name,First Date,Last Date`), 8 069 `-DELISTED` suffixes |
| `ticker_listing?type=etf` | confirmed | 5 100 rows |
| `update_log?type=stock&log_type=delisted` | confirmed | 1 355 rows (`Ticker,Last Date`) |
| `meta_file splits` / `dividends` | confirmed | 302 → zip (Backblaze B2) of per-ticker `{date,ratio}` / `{exdate,amount}` files: 2 963 / 3 900 files |
| `data_file type=stock period=day timeframe=1min adjustment=UNADJUSTED` | confirmed | 302 → `https://x01.frd-data.com/…zip`, 15.76 MB zip, 6 515 files, 66.4 MB raw, no header, `yyyy-MM-dd HH:mm:ss,O,H,L,C,V`, 04:00–19:59 ET (extended hours), bar **open** timestamps |
| `data_file period=full&ticker_range=A-Z` (stock/etf, 1min/5min/30min/1hour/1day, adj_split/adj_splitdiv/UNADJUSTED*) | doc | *UNADJUSTED only 1min + 1day. Bundle size ≈ 34 GB zip / 380 GB raw for stocks+ETFs. **Not downloaded yet** (bounded backfill in Phase 5 only) |
| `full_update period=day|week` (re-adjusted histories) | doc | needed to repair adjusted series after splits/dividends |
| `delisted_data_file archive_number=1..5 / update=week|year` | doc | archives count inconsistent in docs (4 vs 5) |
| futures (continuous ×3 adjustments + individual contracts), index, fx, crypto | confirmed license (last_update) ; doc endpoints | FX intraday = `yyyyMMdd,HH:mm:ss,…` two-field datetime; crypto timestamps UTC; index has no volume; futures daily bars follow the contract trading day |
| options (EOD only, CBOE feed, Greeks) | confirmed license ; doc | no intraday options; bundle only; 70 GB zip incl. options |
| quotes / trades / ticks | **unsupported** | vendor provides bars only |
| Fundamentals `/api-f/v1/financials/all/{ticker}/{annual|quarterly}` | confirmed | AAPL: 14 annual / 59 quarterly records, 76 fields incl. `period_end`, `filing_date`, `fiscal_year`, `fiscal_period`, `actual_year`, `actual_period` |
| `/api-f/v1/financials/single/{ticker}?period=FY|Q1..Q4&year=` | doc | |
| `/api-f/v1/earnings_release/t/{ticker}`, `/calendar/{start}/{end}` | confirmed | AAPL 115 records (`date, eps_estimate, eps_reported, release_time`) |
| `/api-f/v1/metrics/t/{ticker}/{start}/{end}`, `/metrics/last` | confirmed | daily valuation metrics (pe_ttm, market_cap, ev, …) |
| `/api-f/v1/short_interest`, `/short_volume`, `/social_sentiment` | doc | parameter tables only |
| Fundamentals revisions / restatement history | **unknown** | vendor exposes one value per (ticker, period) with `filing_date`; no "as originally reported vs restated" flag → we snapshot each ingestion run and version rows ourselves (see ARCHITECTURE §PIT) |
| Shares outstanding | confirmed (field `shares`, `shares_av`, `shares_av_diluted`) | |
| Corporate actions | confirmed splits + cash dividends only | no mergers/spin-offs |
| Identifier history | partial | ticker changes merged under latest ticker (since 2022), delisted suffixed `-DELISTED`; no CUSIP/ISIN/CIK from vendor → CIK mapping via SEC `company_tickers.json` (connector `sec_edgar`, pending configuration) |
| Rate limits / quotas | none documented ("There are no rate limits") | we still budget 2 concurrent downloads, 1 req/s metadata |
| Pagination | none | whole zip archives |
| Historical depth | stocks from 2000 (ticker listing first dates), options 2010, short data 2021 | |
| Update schedule | docs conflict (23:30 ET / 01:00 ET / 02:00–03:00 ET for full) | policy: poll `last_update` before every incremental job |

Sanitized probe transcripts: `docs/evidence/phase0-probes.md`.

## 4. License (FirstRate Data, `/about/license`, `/about/FAQ`)

- Perpetual private use; data may be stored on own servers, loaded into databases, used for research, backtests, model training, derived
  statistics and charts. Attribution to FirstRate Data required when publishing research.
- **Prohibited**: redistributing raw data to third parties (CSV downloads, an API returning bars, tables of prices/volumes), reselling, sharing
  the user id. Multi-user only within a single institution with a multi-user license.
- Consequence for FinData OS: **owner-only by default**; every API token is scoped; the console shows a license banner; no public routes
  return bars; exports are owner-scoped and expire. The `license_class` column is `frd-private` on every vendor row.

## 5. Capacity model (1-minute data)

Measured: one full trading day of 1-min UNADJUSTED bars for 6 515 stocks = 66.4 MB CSV (15.8 MB zip). Parquet ZSTD(3) with typed
columns compresses FirstRate CSV ≈ 5–6× (prior hfmarketdata lake: 26.5 G rows in 350 GB incl. options → ≈ 13 bytes/row).

| Scenario | Rows | Raw zip kept | Curated Parquet | Total incl. 15 % index/manifests + backup copy ×1 |
|---|---:|---:|---:|---:|
| Stocks, 1-min UNADJUSTED, 1 year incremental (252 d) | ≈ 7.5 × 10⁷ × … ≈ 1.6 × 10⁸ (avg 100 bars/sym/day × 6 500 × 252) | 4 GB | 2.5 GB | ≈ 15 GB |
| Stocks + ETFs, 1-min, full history 2000→2026, one adjustment | ≈ 1.5 × 10¹⁰ | 34 GB (bundle) | ≈ 190 GB | ≈ 500 GB (with backup on M4M64a) |
| + 5-min derived | +20 % | — | +40 GB | |
| Full vendor bundle all timeframes/adjustments (not planned) | ≈ 2.6 × 10¹⁰ | 70 GB | 350 GB | > 800 GB — exceeds single-node budget |

Budget rule adopted: lake ≤ 300 GB on M3U96a (alert at 250 GB, hard stop at 60 GB free), raw zips retained for **incremental** archives
only (immutability proof) and deleted after checksum + Parquet publication for **full** archives (the zip is re-downloadable; its SHA-256 is
kept in the manifest). Full backfill is executed by letter range and by adjustment on explicit operator request (`findata ingest backfill`).

## 6. Existing code and integration points

- `hfmarketdata.git` (spbgit): `frd_downloader.py` (job plan, resume manifest, CSV→Parquet via DuckDB, zip quirks: nested zips, mixed
  LF/CRLF, FX two-field datetime, volume decimals) and `hfmarketdata/api/fundamentals/edgar_client.py` (SEC companyfacts) are reused as
  knowledge; code is rewritten under the FinData contracts (raw immutability, provenance, PIT).
- `mld` (M1M32): deployment, tunnel route, heal. Manifest `M1M32:~/dispatch/apps/findata.json`.
- agentd (M4M64a): auto-discovers PM2 processes and health paths from the registry.
- Postgres 17 local on M3U96a (`findata` role + database created 2026-11-01, localhost only). Redis 8 available (not used in v1: job queue
  is Postgres-backed with `SELECT … FOR UPDATE SKIP LOCKED`, durable across reboots).

## 7. Blocked / deferred (honest list)

- Fundamentals **revision history**: not provided by the vendor; FinData versions rows per ingestion run (`observed_at`) and flags value
  changes as `restatement_flag=true`. Historical restatements before the first ingestion are unknowable from this vendor.
- SEC EDGAR connector: implemented as a modular connector with fixture tests, **pending configuration** (requires a declared User-Agent
  contact e-mail per SEC fair-access policy; no secret).
- Full multi-year 1-min backfill: deferred to Phase 5 (representative letter range), on operator request because of disk budget.
- Options, futures contracts, FX, crypto, index: capability confirmed; connectors implement the file layouts; **ingestion not run** in
  Phase 2 (one licensed dataset end-to-end = US stocks 1-min UNADJUSTED).