# FinData OS — Operations

Node: **M3U96a** (`ssh M3U96a`). App: `~/apps/findata` (git clone, remote-first). Lake: `~/findata-lake`. Secrets: `~/apps/findata/.env` (600; laptop copy `~/Desktop/Cluster/secrets/findata/.env`). Processes (PM2): `findata-api` (:8600), `findata-worker`.

## Daily operation
```bash
ssh M3U96a
cd ~/apps/findata && set -a && . ./.env && set +a
.venv/bin/findata status                    # lake usage, dataset versions, queue
.venv/bin/findata ingest jobs               # last jobs (status, attempts, error)
pm2 logs findata-worker --lines 100         # structured JSON logs (secrets redacted)
curl -s 127.0.0.1:8600/api/v1/health | jq   # alerts (anonymous → status only; use the console for details)
```
Scheduler (inside the worker, America/New_York): incremental `day` polling from 02:00 every 30 min (skips when the vendor's `last_update` is unchanged), Saturday 06:00 `week` repair, Sunday 07:00 metadata + compaction, 05:00 fundamentals (tracked tickers), 04:00 backup, 06:00 quality + gc.

## Manual jobs
```bash
.venv/bin/findata ingest incremental --type stock --period day [--force]
.venv/bin/findata ingest backfill Z --type stock            # one letter archive (bounded; check free space first)
.venv/bin/findata ingest fundamentals AAPL,MSFT,NVDA
.venv/bin/findata ingest derive --dates 2026-10-30
.venv/bin/findata ingest compaction --min-age-days 35
.venv/bin/findata backup && .venv/bin/findata restore-drill
```
Jobs can also be enqueued/retried/cancelled from the console (Ingestion page) or `POST /api/v1/admin/jobs`.

## Deployment (mld, from the laptop)
```bash
~/Desktop/cluster-skill/mld deploy findata        # pinned to M3U96a; runs post_sync hooks (uv sync, web build, migrate)
~/Desktop/cluster-skill/mld status --live
```
Remote-first: edit on the node, `pm2 restart findata-api findata-worker`, then `git commit && git push origin main` (origin = spbgit `findata.git`).

## Backups, RPO / RTO
- Daily 04:00 ET: `pg_dump | zstd` (14 kept) + `rsync -a --delete` of `~/findata-lake` (minus `_tmp`, `_retired`, `exports`) → `M4M64a:~/backups/findata/{lake,pgdump}`; remote file count compared with local (`verified`).
- **RPO**: 24 h for metadata and curated data (raw archives are re-downloadable from the vendor while the subscription is active; sha256 kept). **RTO**: ≈ 30 min for metadata (restore dump, start API) + lake copy time over LAN (≈ 1 Gbit/s → 100 GB in 15–20 min).
- Restore drill: `findata restore-drill --dest /tmp/findata-restore-drill` restores the latest dump into a scratch database, copies one dataset back and verifies every Parquet sha256 against the manifest. Record the output in `docs/STATUS.md`.
- Full restore procedure: (1) `createdb findata && zstd -dc dump.sql.zst | psql findata`; (2) `rsync -a M4M64a:~/backups/findata/lake/ ~/findata-lake/`; (3) `findata migrate`; (4) start PM2 processes; (5) `findata status` must show the same versions as the dump.

## Monitoring
- `GET /metrics` (Prometheus text; allowed from private addresses or admin): `findata_lake_free_gb`, `findata_lake_curated_bytes`, `findata_jobs{status}`, `findata_dataset_freshness_days`, `findata_vendor_lag_days`, `findata_backup_age_hours`, `findata_quality_anomalies_7d`, HTTP counters/latency. Alert rules: `infra/prometheus/findata-alerts.yml`.
- `GET /api/v1/health` (authenticated): SSD SMART status, lake budget, worker lock, dead jobs, staleness vs vendor, backup age.
- agentd (M4M64a) and `mld heal` keep PM2 processes up; the console Overview shows the same alerts.

## Incident playbook
| Symptom | Action |
|---|---|
| `worker_down` alert | `pm2 restart findata-worker`; a second worker is refused by the lake lock (by design) |
| job `dead` | inspect `last_error` (console → Ingestion), fix, **Retry** |
| `disk_low/critical` | stop backfills, run `gc`, review `_retired/` and `raw/`; raise budget only after adding storage |
| vendor 403 / HTML instead of zip | subscription or key issue — check the customer page; never paste the key into chat or code |
| stale dataset | vendor publishes 23:30–03:00 ET; `findata ingest incremental --force` after confirming `last_update` |