FinData OS — STATUS (updated 2026-11-01, build session)
Legend: ✅ done and verified · 🟡 done, not yet verified in production · ⛔ blocked · ⬜ not started
Phase 0 — Discover ✅
- ✅ Cluster/SSD inventory, node choice M3U96a with rationale (
docs/DISCOVERY.md§1) - ✅ Vendor identified (FirstRate Data), auth model, capability matrix, license summary, sanitized probes (
docs/evidence/phase0-probes.md) - ✅ Minimal authorized test requests (last_update ×7 types, ticker listings, one 15.8 MB daily 1-min archive, splits/dividends, fundamentals JSON)
- ✅ Capacity model and disk budget (≤ 250 GB curated, 60 GB hard stop)
Phase 1 — Foundation
- ✅ Monorepo (uv workspace, 8 packages), migrations (
findata migrate), secrets in.envonly, redaction filter - ✅ Authentication: owner password (argon2) + TOTP MFA, sessions, CSRF, scoped API keys; tests green
- 🟡 Deployment via
mldto M3U96a, TLS via MacLustr Tunnel, health checks — see "Production" below - ✅ CI-equivalent:
make test(37 tests),make lint(ruff clean),pip-audit(no known vulnerabilities, 2026-11-01)
Phase 2 — Market pipeline (US stocks, 1-minute)
- ✅ Connector (discover/capabilities/fetch_range/normalize/validate/checkpoint), immutable raw archives, delta Parquet per session, atomic versioned publication, supersede without duplicates (tested), 5-minute derivation (tested vs manual aggregation and DST), coverage + quality anomalies
- 🟡 Real incremental run on the node (see Production)
Phase 3 — Fundamentals
- ✅ FirstRate fundamentals connector (verified live on AAPL during discovery), long-format PIT facts,
available_atpolicy, revision/restatement tracking, as-of query hides later filings and later-observed restatements (tested), earnings events - ⬜ SEC EDGAR connector: parsing implemented + fixture-tested; pending configuration (
SEC_EDGAR_USER_AGENT) - ⚠️ Limitation: vendor gives no revision history → restatements before first ingestion are unknown
Phase 4 — Platform
- ✅ REST API
/api/v1(38+ routes, OpenAPI), cursor pagination, Arrow/CSV/JSON, async Parquet exports, sandboxed SQL, coverage/quality/health/admin, Prometheus metrics - ✅ Python SDK (bars via Arrow pages, exports, fundamentals, SQL) — tested against a live uvicorn thread
- 🟡 Web console (Next.js static export) — built by a sub-agent; see Production for verification
Phase 5 — Scale
- 🟡 Representative multi-year backfill (bounded letter range) — to run on the node
- 🟡 Backup + restore drill — to run on the node
- ⬜ Load benchmark
Phase 6 — AI and polish
- ✅ Assistant endpoint implemented (Anthropic SDK, model proposes SQL only; FinData validates/executes/cites versions) — pending configuration until
ANTHROPIC_API_KEYis set and a capability/cost test is recorded here; OpenAI provider not implemented - ⬜ Security review pass on production
Production evidence (2026-11-01, all outputs real)
Deployment: mld deploy findata --node M3U96a → PM2 findata-api (0.0.0.0:8600 behind a source-network guard) + findata-worker; Caddy route https://findata.spboucher.ai → 10.67.0.10:8600; node dir adopted as git clone (mld adopt, HEAD = origin/main). Public checks: /api/v1/health 200, console / 200, /login/ 200, /api/docs 200; headers HSTS, CSP default-src 'self', X-Frame-Options: DENY, X-Request-ID.
Gotcha fixed during deploy: the API must not bind 127.0.0.1 only (Caddy reaches the node over WireGuard), and --proxy-headers must be off so the guard sees the transport peer (X-Forwarded-For is read only when the peer is the 10.67.0.1 gateway).
Owner login: created (findata create-owner), credentials in the laptop secrets folder; POST /auth/login 200 → /auth/me owner; wrong password 401; POST without CSRF 403; logout 200. TOTP not yet enrolled (owner action in Settings).
Console (headless Chromium against production): redirect to /login/, login, Overview rendered at 375 px (scrollWidth 375, no overflow) and 1280 px dark; 0 console/page errors; license banner present; Bars and Datasets pages render (datasets show unsupported/pending_configuration statuses).
Real ingestion runs (worker logs / job results)
| Job | Result |
|---|---|
| metadata | 16 313 stock + 5 100 ETF listings → 21 243 instruments (13 209 active, 8 069 -DELISTED); 6 131 split rows, 243 209 dividend rows |
| incremental stock day 1min UNADJUSTED | vendor last_update 2026-10-30; archive 15 759 934 B, 6 515 files; 1 337 408 bars published (session 2026-10-30), version 1 |
| derive 5m (session) | 412 938 bars |
| fundamentals AAPL,MSFT,NVDA,JPM,XOM | 24 790 facts (quarterly + annual), earnings events; 7 s |
| backfill Z (period=full, ticker_range=Z, 1min UNADJUSTED) | 57 symbols, 24 815 063 rows, 471 base files, 67 s; 10 570 rows skipped because 2026-10-30 already lives in the delta file (no duplicates); 1 556 anomalies flagged (price_jump 1 480, outside_session 130 — flagged, not altered) |
| lake after backfill | market_bars.stock.1m.raw v58: 472 files, 26 152 471 rows, 331.5 MB; sessions 2000-01-04 → 2026-10-30; 6 521 symbols (6 515 one session + 57 Z full history) |
| backup #1 | pg_dump 3.0 MB zstd + rsync to M4M64a, remote file count verified (ran before the backfill; a second backup follows the compaction) |
Lake: 506.9 GB free, curated 0.54 GB, raw 0.02 GB (full Z archive deleted after publication, sha256 kept).
SDK against production (findata_sdk, API key scoped read:market/read:fundamentals/query/export): health ok (SMART Verified, worker running); AAPL 1m 900 rows (first bar 08:00Z = 04:00 ET open, close 351.6499) in 0.30 s; 5m 192 rows with volume conserved; split_div adjustment applied at query time; fundamentals as_of 2025-01-01 → 156 rows, 536 hidden (latest period 2024-09-28 filed 2024-11-01) vs now → period 2026-06-27; SQL top-volume query 0.08 s; export of 5 symbols → Parquet 4 508 rows / 99 KB in 1.15 s.
Load benchmark (laptop → public URL over the tunnel)
| Request | n / conc | p50 | p95 | rps |
|---|---|---|---|---|
| bars 1m one session (1 000 rows) | 80 / 8 | 232 ms | 800 ms | 28.7 |
| bars 1m 31-day span 2019 (multi-year base file) | 20 / 4 | 1 572 ms | 1 687 ms | 2.9 |
| bars 5m 180-day span | 20 / 4 | 2 214 ms | 2 957 ms | 1.7 |
| SQL per-year aggregate on one symbol (26 M-row lake) | 6 / 2 | 1 567 ms | 1 786 ms | 1.3 |
| SQL count(*) over the whole lake | 3 / 1 | 985 ms | — | — |
| catalog (Postgres) | 100 / 16 | 101 ms | 209 ms | 135 |
Interpretation: file selection is by lake_files date range, so a 31-day 1m request scans the symbol's yearly base file (≈ 100 k rows) then paginates — acceptable; compaction of the 5m deltas (6 748 session files from the Z backfill) into base files was queued to fix the derived dataset's file count. |
Compaction / backup #2 / restore drill / quality report: see "Final checks" below.