spb/findata

Public

46commits
1branches
0releases
2.8 MBon disk
maindefault branch
19 h agolast push

docs: bulk-load incidents and rules

Simon-Pierre Boucher committed yesterday (Nov 2, 2026) parent 3f3ea62 Browse files

2 changed files +9 −0

modified docs/OPERATIONS.md +5 −0

				@@ -48,6 +48,11 @@ Remote-first: edit on the node, `pm2 restart findata-api findata-worker`, then `
48	- `GET /api/v1/health` (authenticated): SSD SMART status, lake budget, worker lock, dead jobs, staleness vs vendor, backup age.	48	- `GET /api/v1/health` (authenticated): SSD SMART status, lake budget, worker lock, dead jobs, staleness vs vendor, backup age.
49	- agentd (M4M64a) and `mld heal` keep PM2 processes up; the console Overview shows the same alerts.	49	- agentd (M4M64a) and `mld heal` keep PM2 processes up; the console Overview shows the same alerts.
50		50
		51	+## Rules learned during the bulk load
		52	+- Health/metrics probes must be O(1): never walk the lake on disk; sizes come from `lake_files`/`raw_archives`; `/health` is served from a 60 s cache. `mld heal` restarts every process of the app when the local probe fails twice.
		53	+- PM2 processes on this node run through `run-under-sshd.sh`; stop/restart is graceful (SIGTERM forwarded, waits for the current job). If a process must be killed immediately: `kill -9` the remote python (pid in `logs/sshd-<name>.pid`), remove the pidfile, `pm2 restart <name>`.
		54	+- Bulk workers: 6–8 is the memory ceiling on 96 GB (each holds 2–10 GB while converting large letters).
		55	+
51	## Incident playbook	56	## Incident playbook
52	\| Symptom \| Action \|	57	\| Symptom \| Action \|
53	\|---\|---\|	58	\|---\|---\|

modified docs/STATUS.md +4 −0

				@@ -101,6 +101,10 @@ Interpretation: file selection is by `lake_files` date range, so a 31-day 1m req
101	- Architecture change: two lake roots (hot internal / cold external), 57 new dataset rows (migration 0002), parsers for index (no volume), futures contracts, option chains; `options` task; 4 bulk workers in parallel; scheduler extended (daily index/FX/crypto/ETF 1m + 1d, weekly futures/contracts/adjusted `full_update`/delisted week, monthly options). Tests: 44 green.	101	- Architecture change: two lake roots (hot internal / cold external), 57 new dataset rows (migration 0002), parsers for index (no volume), futures contracts, option chains; `options` task; 4 bulk workers in parallel; scheduler extended (daily index/FX/crypto/ETF 1m + 1d, weekly futures/contracts/adjusted `full_update`/delisted week, monthly options). Tests: 44 green.
102	- Plan enqueued: 798 jobs (4 running at launch: stock 1min UNADJUSTED letters A–D, ≈ 4 GB zip each) + fundamentals for 13 209 active stocks. Representative sizes measured by HEAD: stock A 1min 3.97 GB (UNADJUSTED) / 4.15 GB (adj_splitdiv), ETF S 1min 1.2 GB, futures continuous 1min 2.4 GB per adjustment, index 1min 1.9 GB, FX 1min 4.5 GB, crypto 1min 2.3 GB, options quarter ≈ 2.3 GB (67 quarters), delisted archive 3.9 GB. Expected: several hundred GB of raw zips on the cold root and ≈ 1 TB of Parquet; several hours to a day of processing.	102	- Plan enqueued: 798 jobs (4 running at launch: stock 1min UNADJUSTED letters A–D, ≈ 4 GB zip each) + fundamentals for 13 209 active stocks. Representative sizes measured by HEAD: stock A 1min 3.97 GB (UNADJUSTED) / 4.15 GB (adj_splitdiv), ETF S 1min 1.2 GB, futures continuous 1min 2.4 GB per adjustment, index 1min 1.9 GB, FX 1min 4.5 GB, crypto 1min 2.3 GB, options quarter ≈ 2.3 GB (67 quarters), delisted archive 3.9 GB. Expected: several hundred GB of raw zips on the cold root and ≈ 1 TB of Parquet; several hours to a day of processing.
103	- Launch incident (fixed): the first jobs failed with `InterruptedError` on `mkdir /Volumes/ssd/…` — macOS TCC blocks external-volume access for PM2-launched processes (node and python have no Full Disk Access; TCC.db is SIP read-only). Fix: all PM2 processes now run through `infra/scripts/run-under-sshd.sh` (sshd has Full Disk Access), Spotlight indexing disabled on the external SSD, EINTR-safe mkdir, graceful SIGHUP. Verified 20:13 ET: four archives downloading to `/Volumes/ssd/findata/raw` at ≈ 80 MB/s aggregate, first letter (Q) converting (1.0 M rows / 4 symbols after 23 s).	103	- Launch incident (fixed): the first jobs failed with `InterruptedError` on `mkdir /Volumes/ssd/…` — macOS TCC blocks external-volume access for PM2-launched processes (node and python have no Full Disk Access; TCC.db is SIP read-only). Fix: all PM2 processes now run through `infra/scripts/run-under-sshd.sh` (sshd has Full Disk Access), Spotlight indexing disabled on the external SSD, EINTR-safe mkdir, graceful SIGHUP. Verified 20:13 ET: four archives downloading to `/Volumes/ssd/findata/raw` at ≈ 80 MB/s aggregate, first letter (Q) converting (1.0 M rows / 4 symbols after 23 s).
		104	+- Incident 2026-11-01 23:50 → 2026-11-02 00:55 ET (API unavailable ~1 h): `/api/v1/health` walked the whole lake on disk (`rglob`, 400 k files on two volumes) and exceeded the 6 s probe timeout of `mld heal`, which then restarted every FinData process every 5 min; uvicorn hung in graceful shutdown behind the slow handler. Fix: lake usage now comes from the catalog (`lake_files`, `raw_archives`) and the health evaluation is cached 60 s; hung processes killed, API back (3 ms). Lesson recorded in OPERATIONS: probes must never depend on filesystem walks.
		105	+- Incident 2026-11-01 ~21:00: per-version manifests with full file lists reached 86 GB on the internal SSD during the bulk load → manifests are now compact deltas; the full list of any version is reconstructed from `lake_files` (`findata manifest <dataset> [--version]`).
		106	+- Incident 2026-11-01 ~21:20: PM2 restarts left orphan workers (ssh wrapper exited, remote kept running) → wrapper now forwards SIGTERM to the remote process and waits; leftovers from a previous incarnation are terminated on start.
		107	+- Progress 2026-11-02 00:45 ET: 187 archives done / 0 dead; stock & ETF 1-min raw A–Z complete (4.29 G + 0.9 G bars), ETF 1-day complete, index/FX/crypto/futures continuous 1-min + 1-day complete, options 61/68 quarters (≈ 3.1 G option rows so far), fundamentals 2 029 / 13 209 tickers (≈ half of "active" listings are warrants/units/CEFs/notes without statements → vendor 404, logged and skipped).
104	- Evidence of completion will be appended below as jobs finish (`findata status`, console Storage/Ingestion).	108	- Evidence of completion will be appended below as jobs finish (`findata status`, console Storage/Ingestion).
105		109
106	## Known limitations / exact next steps	110	## Known limitations / exact next steps
107		111