# FinData OS — Operations

Node: M3U96a (ssh M3U96a). App: ~/apps/findata (git clone, remote-first). Lake: hot root ~/findata-lake (internal SSD: stock/ETF 1m + 1d raw, derived 5m, manifests, exports) and cold root /Volumes/ssd/findata (external 4 TB Thunderbolt SSD, marker .findata-cold-root: raw archives + every other dataset — adjusted series, coarse timeframes, futures, index, FX, crypto, options). If the external SSD is not mounted the workers refuse to start (marker check) rather than writing to the wrong disk. Secrets: ~/apps/findata/.env (600; laptop copy ~/Desktop/Cluster/secrets/findata/.env). Processes (PM2): findata-api (:8600), findata-worker (primary: lake lock, scheduler, incremental/derive/compaction/backup/quality/gc), findata-bulk-1..4 (parallel backfill/options/fundamentals; publication serialized by Postgres row locks; compaction defers while backfills are pending on a dataset).

# macOS privacy (TCC) and the external SSD — read this before touching PM2

Processes started by PM2/launchd on M3U96a have no Full Disk Access; any access to /Volumes/ssd from them blocks forever on a hidden consent prompt (observed: mkdir stuck, then EINTR). sshd-keygen-wrapper does have Full Disk Access, so every FinData process is started through infra/scripts/run-under-sshd.sh, which re-executes the real command on localhost via ssh (key ~/.ssh/id_ed25519_maclustr, no agent). PM2 supervises the ssh client; stopping it hangs up the pty and the worker receives SIGHUP (graceful stop after the current job; kill_timeout 1 h for workers). The TCC database is SIP-protected (read-only), so the clean alternative is manual: System Settings → Privacy & Security → Full Disk Access → add /opt/homebrew/bin/node and the uv Python; then the wrapper can be dropped. Never start a FinData worker directly with pm2 start python … on this node.

# Full vendor download (launched 2026-11-01 evening)

findata ingest plan-all lists the complete plan (798 jobs: 730 archive backfills over 70 datasets + 68 option-chain archives); findata ingest backfill-all --yes enqueues it (refuses without the cold root or with < 1200 GB free there / < 300 GB on the hot root). Priorities: stock/ETF 1-min raw A–Z first, then 1-day raw, index/FX/crypto/futures, delisted archives, vendor-adjusted series, coarse timeframes, options 2010→today. Raw zips are kept on the cold root (sha256 in raw_archives). Progress: console → Ingestion, or findata ingest jobs. Fundamentals for every active stock: findata ingest fundamentals --all-active (checkpointed per ticker).

# Daily operation

ssh M3U96a
cd ~/apps/findata && set -a && . ./.env && set +a
.venv/bin/findata status                    # lake usage, dataset versions, queue
.venv/bin/findata ingest jobs               # last jobs (status, attempts, error)
pm2 logs findata-worker --lines 100         # structured JSON logs (secrets redacted)
curl -s 127.0.0.1:8600/api/v1/health | jq   # alerts (anonymous → status only; use the console for details)

Scheduler (inside the worker, America/New_York): incremental day polling from 02:00 every 30 min (skips when the vendor's last_update is unchanged), Saturday 06:00 week repair, Sunday 07:00 metadata + compaction, 05:00 fundamentals (tracked tickers), 04:00 backup, 06:00 quality + gc.

# Manual jobs

.venv/bin/findata ingest incremental --type stock --period day [--force]
.venv/bin/findata ingest backfill Z --type stock            # one letter archive (bounded; check free space first)
.venv/bin/findata ingest fundamentals AAPL,MSFT,NVDA
.venv/bin/findata ingest derive --dates 2026-10-30
.venv/bin/findata ingest compaction --min-age-days 35
.venv/bin/findata backup && .venv/bin/findata restore-drill

Jobs can also be enqueued/retried/cancelled from the console (Ingestion page) or POST /api/v1/admin/jobs.

# Deployment (mld, from the laptop)

~/Desktop/cluster-skill/mld deploy findata        # pinned to M3U96a; runs post_sync hooks (uv sync, web build, migrate)
~/Desktop/cluster-skill/mld status --live

Remote-first: edit on the node, pm2 restart findata-api findata-worker, then git commit && git push origin main (origin = spbgit findata.git).

# Backups, RPO / RTO

Scope: the daily backup covers the hot root (~/findata-lake: stock/ETF 1m + 1d raw, derived 5m, manifests, exports) and the metadata database. The cold root (/Volumes/ssd/findata: raw vendor archives, adjusted/coarse/futures/index/FX/crypto/options datasets, ≈ 1 TB) is not backed up — it is fully re-downloadable from the vendor while the subscription is active (archive sha256 recorded in raw_archives) and the catalog restore recreates the exact file list. First full-catalog backup 2026-11-02 04:04 → 04:41 (36 min, pg_dump 2.1 GB incl. 18 M fundamentals facts).

• Daily 04:00 ET: pg_dump | zstd (14 kept) + rsync -a --delete of ~/findata-lake (minus _tmp, _retired, exports) → M4M64a:~/backups/findata/{lake,pgdump}; remote file count compared with local (verified).
• RPO: 24 h for metadata and curated data (raw archives are re-downloadable from the vendor while the subscription is active; sha256 kept). RTO: ≈ 30 min for metadata (restore dump, start API) + lake copy time over LAN (≈ 1 Gbit/s → 100 GB in 15–20 min).
• Restore drill: findata restore-drill --dest /tmp/findata-restore-drill restores the latest dump into a scratch database, copies one dataset back and verifies every Parquet sha256 against the manifest. Record the output in docs/STATUS.md.
• Full restore procedure: (1) createdb findata && zstd -dc dump.sql.zst | psql findata; (2) rsync -a M4M64a:~/backups/findata/lake/ ~/findata-lake/; (3) findata migrate; (4) start PM2 processes; (5) findata status must show the same versions as the dump.

# Monitoring

• GET /metrics (Prometheus text; allowed from private addresses or admin): findata_lake_free_gb, findata_lake_curated_bytes, findata_jobs{status}, findata_dataset_freshness_days, findata_vendor_lag_days, findata_backup_age_hours, findata_quality_anomalies_7d, HTTP counters/latency. Alert rules: infra/prometheus/findata-alerts.yml.
• GET /api/v1/health (authenticated): SSD SMART status, lake budget, worker lock, dead jobs, staleness vs vendor, backup age.
• agentd (M4M64a) and mld heal keep PM2 processes up; the console Overview shows the same alerts.

# Rules learned during the bulk load

• Health/metrics probes must be O(1): never walk the lake on disk; sizes come from lake_files/raw_archives; /health is served from a 60 s cache. mld heal restarts every process of the app when the local probe fails twice.
• PM2 processes on this node run through run-under-sshd.sh; stop/restart is graceful (SIGTERM forwarded, waits for the current job). If a process must be killed immediately: kill -9 the remote python (pid in logs/sshd-<name>.pid), remove the pidfile, pm2 restart <name>.
• Bulk workers: 6–8 is the memory ceiling on 96 GB (each holds 2–10 GB while converting large letters).

# Incident playbook