# FinData OS — Operations

Node: M3U96a (ssh M3U96a). App: ~/apps/findata (git clone, remote-first). Lake: ~/findata-lake. Secrets: ~/apps/findata/.env (600; laptop copy ~/Desktop/Cluster/secrets/findata/.env). Processes (PM2): findata-api (:8600), findata-worker.

# Daily operation

ssh M3U96a
cd ~/apps/findata && set -a && . ./.env && set +a
.venv/bin/findata status                    # lake usage, dataset versions, queue
.venv/bin/findata ingest jobs               # last jobs (status, attempts, error)
pm2 logs findata-worker --lines 100         # structured JSON logs (secrets redacted)
curl -s 127.0.0.1:8600/api/v1/health | jq   # alerts (anonymous → status only; use the console for details)

Scheduler (inside the worker, America/New_York): incremental day polling from 02:00 every 30 min (skips when the vendor's last_update is unchanged), Saturday 06:00 week repair, Sunday 07:00 metadata + compaction, 05:00 fundamentals (tracked tickers), 04:00 backup, 06:00 quality + gc.

# Manual jobs

.venv/bin/findata ingest incremental --type stock --period day [--force]
.venv/bin/findata ingest backfill Z --type stock            # one letter archive (bounded; check free space first)
.venv/bin/findata ingest fundamentals AAPL,MSFT,NVDA
.venv/bin/findata ingest derive --dates 2026-10-30
.venv/bin/findata ingest compaction --min-age-days 35
.venv/bin/findata backup && .venv/bin/findata restore-drill

Jobs can also be enqueued/retried/cancelled from the console (Ingestion page) or POST /api/v1/admin/jobs.

# Deployment (mld, from the laptop)

~/Desktop/cluster-skill/mld deploy findata        # pinned to M3U96a; runs post_sync hooks (uv sync, web build, migrate)
~/Desktop/cluster-skill/mld status --live

Remote-first: edit on the node, pm2 restart findata-api findata-worker, then git commit && git push origin main (origin = spbgit findata.git).

# Backups, RPO / RTO

• Daily 04:00 ET: pg_dump | zstd (14 kept) + rsync -a --delete of ~/findata-lake (minus _tmp, _retired, exports) → M4M64a:~/backups/findata/{lake,pgdump}; remote file count compared with local (verified).
• RPO: 24 h for metadata and curated data (raw archives are re-downloadable from the vendor while the subscription is active; sha256 kept). RTO: ≈ 30 min for metadata (restore dump, start API) + lake copy time over LAN (≈ 1 Gbit/s → 100 GB in 15–20 min).
• Restore drill: findata restore-drill --dest /tmp/findata-restore-drill restores the latest dump into a scratch database, copies one dataset back and verifies every Parquet sha256 against the manifest. Record the output in docs/STATUS.md.
• Full restore procedure: (1) createdb findata && zstd -dc dump.sql.zst | psql findata; (2) rsync -a M4M64a:~/backups/findata/lake/ ~/findata-lake/; (3) findata migrate; (4) start PM2 processes; (5) findata status must show the same versions as the dump.

# Monitoring

• GET /metrics (Prometheus text; allowed from private addresses or admin): findata_lake_free_gb, findata_lake_curated_bytes, findata_jobs{status}, findata_dataset_freshness_days, findata_vendor_lag_days, findata_backup_age_hours, findata_quality_anomalies_7d, HTTP counters/latency. Alert rules: infra/prometheus/findata-alerts.yml.
• GET /api/v1/health (authenticated): SSD SMART status, lake budget, worker lock, dead jobs, staleness vs vendor, backup age.
• agentd (M4M64a) and mld heal keep PM2 processes up; the console Overview shows the same alerts.

# Incident playbook