1# FinData OS — Operations23Node: **M3U96a** (`ssh M3U96a`). App: `~/apps/findata` (git clone, remote-first). Lake: `~/findata-lake`. Secrets: `~/apps/findata/.env` (600; laptop copy `~/Desktop/Cluster/secrets/findata/.env`). Processes (PM2): `findata-api` (:8600), `findata-worker`.45## Daily operation6```bash7ssh M3U96a8cd ~/apps/findata && set -a && . ./.env && set +a9.venv/bin/findata status # lake usage, dataset versions, queue10.venv/bin/findata ingest jobs # last jobs (status, attempts, error)11pm2 logs findata-worker --lines 100 # structured JSON logs (secrets redacted)12curl -s 127.0.0.1:8600/api/v1/health | jq # alerts (anonymous → status only; use the console for details)13```14Scheduler (inside the worker, America/New_York): incremental `day` polling from 02:00 every 30 min (skips when the vendor's `last_update` is unchanged), Saturday 06:00 `week` repair, Sunday 07:00 metadata + compaction, 05:00 fundamentals (tracked tickers), 04:00 backup, 06:00 quality + gc.1516## Manual jobs17```bash18.venv/bin/findata ingest incremental --type stock --period day [--force]19.venv/bin/findata ingest backfill Z --type stock # one letter archive (bounded; check free space first)20.venv/bin/findata ingest fundamentals AAPL,MSFT,NVDA21.venv/bin/findata ingest derive --dates 2026-10-3022.venv/bin/findata ingest compaction --min-age-days 3523.venv/bin/findata backup && .venv/bin/findata restore-drill24```25Jobs can also be enqueued/retried/cancelled from the console (Ingestion page) or `POST /api/v1/admin/jobs`.2627## Deployment (mld, from the laptop)28```bash29~/Desktop/cluster-skill/mld deploy findata # pinned to M3U96a; runs post_sync hooks (uv sync, web build, migrate)30~/Desktop/cluster-skill/mld status --live31```32Remote-first: edit on the node, `pm2 restart findata-api findata-worker`, then `git commit && git push origin main` (origin = spbgit `findata.git`).3334## Backups, RPO / RTO35- Daily 04:00 ET: `pg_dump | zstd` (14 kept) + `rsync -a --delete` of `~/findata-lake` (minus `_tmp`, `_retired`, `exports`) → `M4M64a:~/backups/findata/{lake,pgdump}`; remote file count compared with local (`verified`).36- **RPO**: 24 h for metadata and curated data (raw archives are re-downloadable from the vendor while the subscription is active; sha256 kept). **RTO**: ≈ 30 min for metadata (restore dump, start API) + lake copy time over LAN (≈ 1 Gbit/s → 100 GB in 15–20 min).37- Restore drill: `findata restore-drill --dest /tmp/findata-restore-drill` restores the latest dump into a scratch database, copies one dataset back and verifies every Parquet sha256 against the manifest. Record the output in `docs/STATUS.md`.38- Full restore procedure: (1) `createdb findata && zstd -dc dump.sql.zst | psql findata`; (2) `rsync -a M4M64a:~/backups/findata/lake/ ~/findata-lake/`; (3) `findata migrate`; (4) start PM2 processes; (5) `findata status` must show the same versions as the dump.3940## Monitoring41- `GET /metrics` (Prometheus text; allowed from private addresses or admin): `findata_lake_free_gb`, `findata_lake_curated_bytes`, `findata_jobs{status}`, `findata_dataset_freshness_days`, `findata_vendor_lag_days`, `findata_backup_age_hours`, `findata_quality_anomalies_7d`, HTTP counters/latency. Alert rules: `infra/prometheus/findata-alerts.yml`.42- `GET /api/v1/health` (authenticated): SSD SMART status, lake budget, worker lock, dead jobs, staleness vs vendor, backup age.43- agentd (M4M64a) and `mld heal` keep PM2 processes up; the console Overview shows the same alerts.4445## Incident playbook46| Symptom | Action |47|---|---|48| `worker_down` alert | `pm2 restart findata-worker`; a second worker is refused by the lake lock (by design) |49| job `dead` | inspect `last_error` (console → Ingestion), fix, **Retry** |50| `disk_low/critical` | stop backfills, run `gc`, review `_retired/` and `raw/`; raise budget only after adding storage |51| vendor 403 / HTML instead of zip | subscription or key issue — check the customer page; never paste the key into chat or code |52| stale dataset | vendor publishes 23:30–03:00 ET; `findata ingest incremental --force` after confirming `last_update` |53