1# FinData OS — STATUS (updated 2026-11-01, build session)23Legend: ✅ done and verified · 🟡 done, not yet verified in production · ⛔ blocked · ⬜ not started45## Phase 0 — Discover ✅6- ✅ Cluster/SSD inventory, node choice M3U96a with rationale (`docs/DISCOVERY.md` §1)7- ✅ Vendor identified (FirstRate Data), auth model, capability matrix, license summary, sanitized probes (`docs/evidence/phase0-probes.md`)8- ✅ Minimal authorized test requests (last_update ×7 types, ticker listings, one 15.8 MB daily 1-min archive, splits/dividends, fundamentals JSON)9- ✅ Capacity model and disk budget (≤ 250 GB curated, 60 GB hard stop)1011## Phase 1 — Foundation12- ✅ Monorepo (uv workspace, 8 packages), migrations (`findata migrate`), secrets in `.env` only, redaction filter13- ✅ Authentication: owner password (argon2) + TOTP MFA, sessions, CSRF, scoped API keys; tests green14- 🟡 Deployment via `mld` to M3U96a, TLS via MacLustr Tunnel, health checks — see "Production" below15- ✅ CI-equivalent: `make test` (37 tests), `make lint` (ruff clean), `pip-audit` (no known vulnerabilities, 2026-11-01)1617## Phase 2 — Market pipeline (US stocks, 1-minute)18- ✅ Connector (discover/capabilities/fetch_range/normalize/validate/checkpoint), immutable raw archives, delta Parquet per session, atomic versioned publication, supersede without duplicates (tested), 5-minute derivation (tested vs manual aggregation and DST), coverage + quality anomalies19- 🟡 Real incremental run on the node (see Production)2021## Phase 3 — Fundamentals22- ✅ FirstRate fundamentals connector (verified live on AAPL during discovery), long-format PIT facts, `available_at` policy, revision/restatement tracking, as-of query hides later filings and later-observed restatements (tested), earnings events23- ⬜ SEC EDGAR connector: parsing implemented + fixture-tested; **pending configuration** (`SEC_EDGAR_USER_AGENT`)24- ⚠️ Limitation: vendor gives no revision history → restatements before first ingestion are unknown2526## Phase 4 — Platform27- ✅ REST API `/api/v1` (38+ routes, OpenAPI), cursor pagination, Arrow/CSV/JSON, async Parquet exports, sandboxed SQL, coverage/quality/health/admin, Prometheus metrics28- ✅ Python SDK (bars via Arrow pages, exports, fundamentals, SQL) — tested against a live uvicorn thread29- 🟡 Web console (Next.js static export) — built by a sub-agent; see Production for verification3031## Phase 5 — Scale32- 🟡 Representative multi-year backfill (bounded letter range) — to run on the node33- 🟡 Backup + restore drill — to run on the node34- ⬜ Load benchmark3536## Phase 6 — AI and polish37- ✅ Assistant endpoint implemented (Anthropic SDK, model proposes SQL only; FinData validates/executes/cites versions) — **pending configuration** until `ANTHROPIC_API_KEY` is set and a capability/cost test is recorded here; OpenAI provider not implemented38- ⬜ Security review pass on production3940## Production evidence (2026-11-01, all outputs real)4142**Deployment**: `mld deploy findata --node M3U96a` → PM2 `findata-api` (0.0.0.0:8600 behind a source-network guard) + `findata-worker`; Caddy route `https://findata.spboucher.ai → 10.67.0.10:8600`; node dir adopted as git clone (`mld adopt`, HEAD = origin/main). Public checks: `/api/v1/health` 200, console `/` 200, `/login/` 200, `/api/docs` 200; headers HSTS, CSP `default-src 'self'`, `X-Frame-Options: DENY`, `X-Request-ID`.43Gotcha fixed during deploy: the API must not bind 127.0.0.1 only (Caddy reaches the node over WireGuard), and `--proxy-headers` must be off so the guard sees the transport peer (X-Forwarded-For is read only when the peer is the 10.67.0.1 gateway).4445**Owner login**: created (`findata create-owner`), credentials in the laptop secrets folder; `POST /auth/login` 200 → `/auth/me` owner; wrong password 401; POST without CSRF 403; logout 200. TOTP not yet enrolled (owner action in Settings).4647**Console (headless Chromium against production)**: redirect to `/login/`, login, Overview rendered at 375 px (scrollWidth 375, no overflow) and 1280 px dark; 0 console/page errors; license banner present; Bars and Datasets pages render (datasets show `unsupported`/`pending_configuration` statuses).4849**Real ingestion runs (worker logs / job results)**50| Job | Result |51|---|---|52| metadata | 16 313 stock + 5 100 ETF listings → 21 243 instruments (13 209 active, 8 069 `-DELISTED`); 6 131 split rows, 243 209 dividend rows |53| incremental stock day 1min UNADJUSTED | vendor last_update 2026-10-30; archive 15 759 934 B, 6 515 files; 1 337 408 bars published (session 2026-10-30), version 1 |54| derive 5m (session) | 412 938 bars |55| fundamentals AAPL,MSFT,NVDA,JPM,XOM | 24 790 facts (quarterly + annual), earnings events; 7 s |56| backfill Z (period=full, ticker_range=Z, 1min UNADJUSTED) | 57 symbols, 24 815 063 rows, 471 base files, 67 s; 10 570 rows skipped because 2026-10-30 already lives in the delta file (no duplicates); 1 556 anomalies flagged (price_jump 1 480, outside_session 130 — flagged, not altered) |57| lake after backfill | `market_bars.stock.1m.raw` v58: 472 files, **26 152 471 rows**, 331.5 MB; sessions 2000-01-04 → 2026-10-30; 6 521 symbols (6 515 one session + 57 Z full history) |58| backup #1 | pg_dump 3.0 MB zstd + rsync to M4M64a, remote file count verified (ran before the backfill; a second backup follows the compaction) |5960Lake: 506.9 GB free, curated 0.54 GB, raw 0.02 GB (full Z archive deleted after publication, sha256 kept).6162**SDK against production** (`findata_sdk`, API key scoped read:market/read:fundamentals/query/export): health ok (SMART Verified, worker running); AAPL 1m 900 rows (first bar 08:00Z = 04:00 ET open, close 351.6499) in 0.30 s; 5m 192 rows with volume conserved; `split_div` adjustment applied at query time; fundamentals as_of 2025-01-01 → 156 rows, 536 hidden (latest period 2024-09-28 filed 2024-11-01) vs now → period 2026-06-27; SQL top-volume query 0.08 s; export of 5 symbols → Parquet 4 508 rows / 99 KB in 1.15 s.6364**Load benchmark (laptop → public URL over the tunnel)**65| Request | n / conc | p50 | p95 | rps |66|---|---|---|---|---|67| bars 1m one session (1 000 rows) | 80 / 8 | 232 ms | 800 ms | 28.7 |68| bars 1m 31-day span 2019 (multi-year base file) | 20 / 4 | 1 572 ms | 1 687 ms | 2.9 |69| bars 5m 180-day span | 20 / 4 | 2 214 ms | 2 957 ms | 1.7 |70| SQL per-year aggregate on one symbol (26 M-row lake) | 6 / 2 | 1 567 ms | 1 786 ms | 1.3 |71| SQL count(*) over the whole lake | 3 / 1 | 985 ms | — | — |72| catalog (Postgres) | 100 / 16 | 101 ms | 209 ms | 135 |73Interpretation: file selection is by `lake_files` date range, so a 31-day 1m request scans the symbol's yearly base file (≈ 100 k rows) then paginates — acceptable; compaction of the 5m deltas (6 748 session files from the Z backfill) into base files was queued to fix the derived dataset's file count.7475**Compaction / backup #2 / restore drill / quality report**: see "Final checks" below.76