SPB Git
spboucher.ai

spb/findata

Public

docs: full-load completion report, query performance, SQL rules

Simon-Pierre Boucher committed (Nov 2, 2026) parent a5f98ef Browse files

2 changed files +32 −1

2 changed files
Unified Split
modified docs/API.md +6 −0
@@ -24,3 +24,9 @@ facts = fd.fundamentals("AAPL", as_of="2025-01-01T00:00:00Z", metrics=["rev", "e
24 ``` 24 ```
25 25
26 Limits: 1m bars ≤ 31 days per call (5m ≤ 186 d, 1d ≤ 10 y), ≤ 10 000 rows per page → use exports for larger ranges. Query playground: one read-only statement, 30 s, ≤ 100 000 rows. Rate limit 600 req/min per key (configurable per key). License: FirstRate Data private use — do not redistribute raw data. 26 Limits: 1m bars ≤ 31 days per call (5m ≤ 186 d, 1d ≤ 10 y), ≤ 10 000 rows per page → use exports for larger ranges. Query playground: one read-only statement, 30 s, ≤ 100 000 rows. Rate limit 600 req/min per key (configurable per key). License: FirstRate Data private use — do not redistribute raw data.
27+
28+
29+## SQL playground performance rules (full lake)
30+- Always filter on `symbol = 'X'` / `symbol IN (...)` (bars) or `underlying = 'X'` (options), optionally `year = 2024` or a `session_date` / `trade_date` range: the file list is then resolved from the catalog in milliseconds (`meta.views_used` shows the view name; `(full scan)` means the slow directory path was used).
31+- Cross-sectional questions (all symbols on a date): use `market_bars_stock_1d_raw` (1–2 s) or a `session_date =` predicate on 1m (≈ 3 s).
32+- Views carry the partition columns `symbol`, `year` (bars) and `underlying`, `period` (options) in addition to the data columns.
modified docs/STATUS.md +26 −1
@@ -105,7 +105,32 @@ Interpretation: file selection is by `lake_files` date range, so a 31-day 1m req
105 - Incident 2026-11-01 ~21:00: per-version manifests with full file lists reached 86 GB on the internal SSD during the bulk load → manifests are now compact deltas; the full list of any version is reconstructed from `lake_files` (`findata manifest <dataset> [--version]`). 105 - Incident 2026-11-01 ~21:00: per-version manifests with full file lists reached 86 GB on the internal SSD during the bulk load → manifests are now compact deltas; the full list of any version is reconstructed from `lake_files` (`findata manifest <dataset> [--version]`).
106 - Incident 2026-11-01 ~21:20: PM2 restarts left orphan workers (ssh wrapper exited, remote kept running) → wrapper now forwards SIGTERM to the remote process and waits; leftovers from a previous incarnation are terminated on start. 106 - Incident 2026-11-01 ~21:20: PM2 restarts left orphan workers (ssh wrapper exited, remote kept running) → wrapper now forwards SIGTERM to the remote process and waits; leftovers from a previous incarnation are terminated on start.
107 - Progress 2026-11-02 00:45 ET: 187 archives done / 0 dead; stock & ETF 1-min raw A–Z complete (4.29 G + 0.9 G bars), ETF 1-day complete, index/FX/crypto/futures continuous 1-min + 1-day complete, options 61/68 quarters (≈ 3.1 G option rows so far), fundamentals 2 029 / 13 209 tickers (≈ half of "active" listings are warrants/units/CEFs/notes without statements → vendor 404, logged and skipped). 107 - Progress 2026-11-02 00:45 ET: 187 archives done / 0 dead; stock & ETF 1-min raw A–Z complete (4.29 G + 0.9 G bars), ETF 1-day complete, index/FX/crypto/futures continuous 1-min + 1-day complete, options 61/68 quarters (≈ 3.1 G option rows so far), fundamentals 2 029 / 13 209 tickers (≈ half of "active" listings are warrants/units/CEFs/notes without statements → vendor 404, logged and skipped).
108−- Evidence of completion will be appended below as jobs finish (`findata status`, console Storage/Ingestion). 108+- **Completed 2026-11-02 ≈ 11:20 ET**: 798 / 798 archives succeeded (0 skipped, 0 unparseable files), plus fundamentals for every active stock. Totals (catalog): **564.7 GB curated Parquet, 38.81 G rows, 3.13 M files, 64 active datasets**. Raw archives kept on the external SSD: 806 files, 434 GB. Coverage: stocks 16 195 symbols (incl. delisted) 2000-01-03 → 2026-10-30; ETFs 5 161; indices 122 (2008→); options 11 582 underlyings 2010-01-04 → 2026-10-30 (4.23 G rows); futures 3 continuous series × 5 timeframes + individual contracts; FX, crypto. Fundamentals: 5 451 entities, 18.07 M PIT facts (periods 2010-12 → 2026-10), 373 326 earnings events; 2 658 active listings have no statements (warrants/units/CEFs/notes → vendor 404, recorded in the job checkpoint).
109+- Families (GB / rows): stocks 370.1 / 27.75 G · options 99.2 / 4.23 G · ETFs 57.7 / 3.70 G · futures continuous 12.6 / 1.21 G · futures contracts 10.4 / 0.90 G · FX 8.0 / 0.52 G · crypto 3.7 / 0.23 G · indices 3.2 / 0.26 G. Disks after load: internal 185 GB used (319 GB free after the coverage prune), external 895 GB used (2.98 TB free). Postgres 34 GB (coverage pruned 574 M → 113 M rows, 131 → 25 GB; `coverage` is now kept for the 7 primary datasets only).
110+
111+## Query performance on the full lake (2026-11-02, laptop → public URL over the tunnel, warm)
112+| Request | Result |
113+|---|---|
114+| bars AAPL 1m one day | 55–124 ms (744 rows) |
115+| bars AAPL 1m 31 days (2024-03) | 370–530 ms (15 907 rows) |
116+| bars MSFT 1m 31 days (2008-10) | 0.9–1.6 s (12 918 rows) |
117+| bars AAPL 5m 6 months | 1.0–1.6 s (23 012 rows) |
118+| bars AAPL 1d 10 years | 170–260 ms (2 516 rows) |
119+| bars NVDA 1m 31 days with query-time split+dividend adjustment | 570–780 ms |
120+| bars TWTR-DELISTED 1m 2022-10 | 270–350 ms |
121+| SQL AAPL daily OHLC from 1m, year 2024 | 185 ms |
122+| SQL SPY option chain one day / one expiry | 98 ms (372 rows) · SPY full history count 254 ms (26.6 M rows) |
123+| SQL EURUSD 1m 2020 count · ES ratio-adjusted yearly counts | 30 ms · 52 ms |
124+| SQL 3 symbols split_div daily closes March 2020 | 47 ms |
125+| SQL cross-section: all stocks 1d on one date · all stocks 1m 09:30 bar on one date | 1.3 s (5 962 symbols) · 2.9 s (5 729 symbols) |
126+| export 5 symbols × full year 2024 1m → Parquet | 1.9 s (914 489 rows, 17 MB) |
127+| fundamentals MSFT as-of 2015 · instrument search | 21 ms · 41 ms |
128+How: the API never globs the lake — file lists come from the catalog (`lake_files`, indexed by dataset/date and dataset/symbol); the SQL playground/assistant extract symbol / underlying / year / date predicates from the statement and build the same explicit file list (fast path); statements without any such predicate fall back to a Hive directory scan (30–60 s to list a 170 k–320 k-file dataset) and are documented as slow.
129+
130+## Known limitations after the full load
131+- Whole-dataset SQL scans without a symbol or date predicate are slow (directory listing of up to 320 k files); use predicates, the `*_1d_*` datasets for cross-sections, or exports.
132+- The cold root (external SSD, ≈ 1.3 TB incl. raw archives) is not in the nightly backup (re-downloadable; catalog restore recreates the file list). The hot root + Postgres are backed up nightly (first full-catalog backup 04:04 → 04:41).
133+- `_retired/` holds files superseded during the load (duplicate processing after worker restarts); GC removes them after 7 days.
109 134
110 ## Known limitations / exact next steps 135 ## Known limitations / exact next steps
111 1. **TOTP**: enrol the owner's authenticator in Settings → Security (one click + code). Until then login is password-only. 136 1. **TOTP**: enrol the owner's authenticator in Settings → Security (one click + code). Until then login is password-only.
112 137
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere