1# FinData OS — Operations23Node: **M3U96a** (`ssh M3U96a`). App: `~/apps/findata` (git clone, remote-first). Lake: **hot root** `~/findata-lake` (internal SSD: stock/ETF 1m + 1d raw, derived 5m, manifests, exports) and **cold root** `/Volumes/ssd/findata` (external 4 TB Thunderbolt SSD, marker `.findata-cold-root`: raw archives + every other dataset — adjusted series, coarse timeframes, futures, index, FX, crypto, options). If the external SSD is not mounted the workers refuse to start (marker check) rather than writing to the wrong disk. Secrets: `~/apps/findata/.env` (600; laptop copy `~/Desktop/Cluster/secrets/findata/.env`). Processes (PM2): `findata-api` (:8600), `findata-worker` (primary: lake lock, scheduler, incremental/derive/compaction/backup/quality/gc), `findata-bulk-1..4` (parallel backfill/options/fundamentals; publication serialized by Postgres row locks; compaction defers while backfills are pending on a dataset).45## macOS privacy (TCC) and the external SSD — read this before touching PM26Processes started by PM2/launchd on M3U96a have **no Full Disk Access**; any access to `/Volumes/ssd` from them blocks forever on a hidden consent prompt (observed: `mkdir` stuck, then `EINTR`). `sshd-keygen-wrapper` *does* have Full Disk Access, so every FinData process is started through `infra/scripts/run-under-sshd.sh`, which re-executes the real command on `localhost` via ssh (key `~/.ssh/id_ed25519_maclustr`, no agent). PM2 supervises the ssh client; stopping it hangs up the pty and the worker receives SIGHUP (graceful stop after the current job; `kill_timeout` 1 h for workers). The TCC database is SIP-protected (read-only), so the clean alternative is manual: System Settings → Privacy & Security → Full Disk Access → add `/opt/homebrew/bin/node` and the uv Python; then the wrapper can be dropped. Never start a FinData worker directly with `pm2 start python …` on this node.78## Full vendor download (launched 2026-11-01 evening)9`findata ingest plan-all` lists the complete plan (798 jobs: 730 archive backfills over 70 datasets + 68 option-chain archives); `findata ingest backfill-all --yes` enqueues it (refuses without the cold root or with < 1200 GB free there / < 300 GB on the hot root). Priorities: stock/ETF 1-min raw A–Z first, then 1-day raw, index/FX/crypto/futures, delisted archives, vendor-adjusted series, coarse timeframes, options 2010→today. Raw zips are kept on the cold root (sha256 in `raw_archives`). Progress: console → Ingestion, or `findata ingest jobs`. Fundamentals for every active stock: `findata ingest fundamentals --all-active` (checkpointed per ticker).1011## Daily operation12```bash13ssh M3U96a14cd ~/apps/findata && set -a && . ./.env && set +a15.venv/bin/findata status # lake usage, dataset versions, queue16.venv/bin/findata ingest jobs # last jobs (status, attempts, error)17pm2 logs findata-worker --lines 100 # structured JSON logs (secrets redacted)18curl -s 127.0.0.1:8600/api/v1/health | jq # alerts (anonymous → status only; use the console for details)19```20Scheduler (inside the worker, America/New_York): incremental `day` polling from 02:00 every 30 min (skips when the vendor's `last_update` is unchanged), Saturday 06:00 `week` repair, Sunday 07:00 metadata + compaction, 05:00 fundamentals (tracked tickers), 04:00 backup, 06:00 quality + gc.2122## Manual jobs23```bash24.venv/bin/findata ingest incremental --type stock --period day [--force]25.venv/bin/findata ingest backfill Z --type stock # one letter archive (bounded; check free space first)26.venv/bin/findata ingest fundamentals AAPL,MSFT,NVDA27.venv/bin/findata ingest derive --dates 2026-10-3028.venv/bin/findata ingest compaction --min-age-days 3529.venv/bin/findata backup && .venv/bin/findata restore-drill30```31Jobs can also be enqueued/retried/cancelled from the console (Ingestion page) or `POST /api/v1/admin/jobs`.3233## Deployment (mld, from the laptop)34```bash35~/Desktop/cluster-skill/mld deploy findata # pinned to M3U96a; runs post_sync hooks (uv sync, web build, migrate)36~/Desktop/cluster-skill/mld status --live37```38Remote-first: edit on the node, `pm2 restart findata-api findata-worker`, then `git commit && git push origin main` (origin = spbgit `findata.git`).3940## Backups, RPO / RTO41**Scope**: the daily backup covers the **hot root** (`~/findata-lake`: stock/ETF 1m + 1d raw, derived 5m, manifests, exports) and the metadata database. The **cold root** (`/Volumes/ssd/findata`: raw vendor archives, adjusted/coarse/futures/index/FX/crypto/options datasets, ≈ 1 TB) is **not** backed up — it is fully re-downloadable from the vendor while the subscription is active (archive sha256 recorded in `raw_archives`) and the catalog restore recreates the exact file list. First full-catalog backup 2026-11-02 04:04 → 04:41 (36 min, pg_dump 2.1 GB incl. 18 M fundamentals facts).42- Daily 04:00 ET: `pg_dump | zstd` (14 kept) + `rsync -a --delete` of `~/findata-lake` (minus `_tmp`, `_retired`, `exports`) → `M4M64a:~/backups/findata/{lake,pgdump}`; remote file count compared with local (`verified`).43- **RPO**: 24 h for metadata and curated data (raw archives are re-downloadable from the vendor while the subscription is active; sha256 kept). **RTO**: ≈ 30 min for metadata (restore dump, start API) + lake copy time over LAN (≈ 1 Gbit/s → 100 GB in 15–20 min).44- Restore drill: `findata restore-drill --dest /tmp/findata-restore-drill` restores the latest dump into a scratch database, copies one dataset back and verifies every Parquet sha256 against the manifest. Record the output in `docs/STATUS.md`.45- Full restore procedure: (1) `createdb findata && zstd -dc dump.sql.zst | psql findata`; (2) `rsync -a M4M64a:~/backups/findata/lake/ ~/findata-lake/`; (3) `findata migrate`; (4) start PM2 processes; (5) `findata status` must show the same versions as the dump.4647## Monitoring48- `GET /metrics` (Prometheus text; allowed from private addresses or admin): `findata_lake_free_gb`, `findata_lake_curated_bytes`, `findata_jobs{status}`, `findata_dataset_freshness_days`, `findata_vendor_lag_days`, `findata_backup_age_hours`, `findata_quality_anomalies_7d`, HTTP counters/latency. Alert rules: `infra/prometheus/findata-alerts.yml`.49- `GET /api/v1/health` (authenticated): SSD SMART status, lake budget, worker lock, dead jobs, staleness vs vendor, backup age.50- agentd (M4M64a) and `mld heal` keep PM2 processes up; the console Overview shows the same alerts.5152## Rules learned during the bulk load53- Health/metrics probes must be O(1): never walk the lake on disk; sizes come from `lake_files`/`raw_archives`; `/health` is served from a 60 s cache. `mld heal` restarts *every* process of the app when the local probe fails twice.54- PM2 processes on this node run through `run-under-sshd.sh`; stop/restart is graceful (SIGTERM forwarded, waits for the current job). If a process must be killed immediately: `kill -9` the remote python (pid in `logs/sshd-<name>.pid`), remove the pidfile, `pm2 restart <name>`.55- Bulk workers: 6–8 is the memory ceiling on 96 GB (each holds 2–10 GB while converting large letters).5657## Incident playbook58| Symptom | Action |59|---|---|60| `worker_down` alert | `pm2 restart findata-worker`; a second worker is refused by the lake lock (by design) |61| job `dead` | inspect `last_error` (console → Ingestion), fix, **Retry** |62| `disk_low/critical` | stop backfills, run `gc`, review `_retired/` and `raw/`; raise budget only after adding storage |63| vendor 403 / HTML instead of zip | subscription or key issue — check the customer page; never paste the key into chat or code |64| stale dataset | vendor publishes 23:30–03:00 ET; `findata ingest incremental --force` after confirming `last_update` |65