spb/findata

Public

46commits
1branches
0releases
2.8 MBon disk
maindefault branch
yesterdaylast push

docs: cold root, bulk workers, full download plan

Simon-Pierre Boucher committed yesterday (Nov 2, 2026) parent 17d97ad Browse files

2 changed files +10 −1

modified docs/OPERATIONS.md +4 −1

				`@@ -1,6 +1,9 @@`
1	# FinData OS — Operations	1	# FinData OS — Operations
2		2
3	−Node: M3U96a (`ssh M3U96a`). App: `~/apps/findata` (git clone, remote-first). Lake: `~/findata-lake`. Secrets: `~/apps/findata/.env` (600; laptop copy `~/Desktop/Cluster/secrets/findata/.env`). Processes (PM2): `findata-api` (:8600), `findata-worker`.	3	+Node: M3U96a (`ssh M3U96a`). App: `~/apps/findata` (git clone, remote-first). Lake: hot root `~/findata-lake` (internal SSD: stock/ETF 1m + 1d raw, derived 5m, manifests, exports) and cold root `/Volumes/ssd/findata` (external 4 TB Thunderbolt SSD, marker `.findata-cold-root`: raw archives + every other dataset — adjusted series, coarse timeframes, futures, index, FX, crypto, options). If the external SSD is not mounted the workers refuse to start (marker check) rather than writing to the wrong disk. Secrets: `~/apps/findata/.env` (600; laptop copy `~/Desktop/Cluster/secrets/findata/.env`). Processes (PM2): `findata-api` (:8600), `findata-worker` (primary: lake lock, scheduler, incremental/derive/compaction/backup/quality/gc), `findata-bulk-1..4` (parallel backfill/options/fundamentals; publication serialized by Postgres row locks; compaction defers while backfills are pending on a dataset).
		4	+
		5	+## Full vendor download (launched 2026-11-01 evening)
		6	+`findata ingest plan-all` lists the complete plan (798 jobs: 730 archive backfills over 70 datasets + 68 option-chain archives); `findata ingest backfill-all --yes` enqueues it (refuses without the cold root or with < 1200 GB free there / < 300 GB on the hot root). Priorities: stock/ETF 1-min raw A–Z first, then 1-day raw, index/FX/crypto/futures, delisted archives, vendor-adjusted series, coarse timeframes, options 2010→today. Raw zips are kept on the cold root (sha256 in `raw_archives`). Progress: console → Ingestion, or `findata ingest jobs`. Fundamentals for every active stock: `findata ingest fundamentals --all-active` (checkpointed per ticker).
4		7
5	## Daily operation	8	## Daily operation
6	```bash	9	```bash

modified docs/STATUS.md +6 −0

				@@ -96,6 +96,12 @@ Interpretation: file selection is by `lake_files` date range, so a 31-day 1m req
96	\| 9 \| Licensing and unsupported datasets surfaced in UI and docs \| ✅ license banner, dataset statuses `unsupported` (ticks) / `pending_configuration` (SEC EDGAR) / `planned` \|	96	\| 9 \| Licensing and unsupported datasets surfaced in UI and docs \| ✅ license banner, dataset statuses `unsupported` (ticks) / `pending_configuration` (SEC EDGAR) / `planned` \|
97	\| 10 \| Final operator report \| ✅ this file + `docs/DISCOVERY.md`, `docs/OPERATIONS.md` \|	97	\| 10 \| Final operator report \| ✅ this file + `docs/DISCOVERY.md`, `docs/OPERATIONS.md` \|
98		98
		99	+## Full FirstRate download (started 2026-11-01 ~19:55 ET)
		100	+- Owner request: download the entire licensed catalog, using the internal SSD and the external 4 TB SSD (`/Volumes/ssd`, empty, APFS, 1.9 GB/s write). Owner approved moving `UQO_RESEARCH_DB/data` (32 GB) and `backups/autoscholar` (19 GB) to the external SSD (done, symlinks left; internal free 507 → 560 GB) and declined deleting the retired `rareindex` Postgres DB (179 GB), AI model caches (59 GB) and old git copies (21 GB); production apps stay on M3U96a.
		101	+- Architecture change: two lake roots (hot internal / cold external), 57 new dataset rows (migration 0002), parsers for index (no volume), futures contracts, option chains; `options` task; 4 bulk workers in parallel; scheduler extended (daily index/FX/crypto/ETF 1m + 1d, weekly futures/contracts/adjusted `full_update`/delisted week, monthly options). Tests: 44 green.
		102	+- Plan enqueued: 798 jobs (4 running at launch: stock 1min UNADJUSTED letters A–D, ≈ 4 GB zip each) + fundamentals for 13 209 active stocks. Representative sizes measured by HEAD: stock A 1min 3.97 GB (UNADJUSTED) / 4.15 GB (adj_splitdiv), ETF S 1min 1.2 GB, futures continuous 1min 2.4 GB per adjustment, index 1min 1.9 GB, FX 1min 4.5 GB, crypto 1min 2.3 GB, options quarter ≈ 2.3 GB (67 quarters), delisted archive 3.9 GB. Expected: several hundred GB of raw zips on the cold root and ≈ 1 TB of Parquet; several hours to a day of processing.
		103	+- Evidence of completion will be appended below as jobs finish (`findata status`, console Storage/Ingestion).
		104	+
99	## Known limitations / exact next steps	105	## Known limitations / exact next steps
100	1. TOTP: enrol the owner's authenticator in Settings → Security (one click + code). Until then login is password-only.	106	1. TOTP: enrol the owner's authenticator in Settings → Security (one click + code). Until then login is password-only.
101	2. Coverage: production holds one full-market session (2026-10-30) + full history for the 57 Z tickers. Extend with `findata ingest backfill <letters>` (≈ 258 MB zip and 67 s per small letter; large letters such as A/S take several GB and tens of minutes; budget guard stops below 60 GB free). Daily incremental runs automatically from 02:00 ET.	107	2. Coverage: production holds one full-market session (2026-10-30) + full history for the 57 Z tickers. Extend with `findata ingest backfill <letters>` (≈ 258 MB zip and 67 s per small letter; large letters such as A/S take several GB and tens of minutes; budget guard stops below 60 GB free). Daily incremental runs automatically from 02:00 ET.
102		108