spb/market-atlas

1# Deploying Market Atlas on MacLustr23> **Since 2026-09-13 production runs on the OVH server BHS128** (`ssh BHS128`, ubuntu@51.161.112.69, Ubuntu 24.04, 12 threads / 128 GB),4> outside `mld`: code in `~/apps/market-atlas`, Postgres 17 native (db/role `market_atlas`, en_US.UTF-8), data in `~/market-atlas-data`,5> PM2 under systemd (`pm2-ubuntu`, `LimitNOFILE` 1 048 576) with `~/apps/pm2.market-atlas.config.cjs` generated from the mld manifest by6> `~/apps/.manifests/gen-pm2.py` (secrets, 0600). Tunnel route: `mlt add www.market-atlas.co BHS128:8380 --websocket` (BHS64 Caddy → wg1 10.67.0.60).7> Firewall: `ufw allow in on wg1 from 10.67.0.1 to any port 8380 proto tcp`. The apps were moved because the LAN uplink (Bell) could not8> sustain the cluster's outbound connection count. **Release on BHS128**: `rsync -az --exclude node_modules --exclude .git --exclude 'apps/web/.next' --exclude 'apps/api/dist' --exclude data . BHS128:apps/market-atlas/`9> then `ssh BHS128 'cd apps/market-atlas && pnpm install --frozen-lockfile && pnpm --filter @market-atlas/api build && (cd apps/web && API_URL=http://127.0.0.1:8380 NEXT_PUBLIC_SITE_URL=https://www.market-atlas.co pnpm build) && pm2 restart market-atlas-core market-atlas-web && pm2 save'`.10> pnpm ≥ 10 needs `onlyBuiltDependencies` (esbuild) in `pnpm-workspace.yaml`, otherwise `ERR_PNPM_IGNORED_BUILDS` on a fresh install.11> The M2U64 copy (`~/apps/market-atlas`, `~/market-atlas-data`, Postgres `market_atlas`) was deleted on 2026-09-14: BHS128 holds the only production12> data — schedule an off-host copy of `~/market-atlas-data/backups` and `archive/`. The `mld` procedure below is kept for reference (manifest backup:13> laptop `~/Desktop/Cluster/secrets/mld-manifests-retired-20260914/market-atlas.json`).1415Production formerly ran on **M2U64** (Mac Studio M2 Ultra, 24 c / 64 GB) behind the MacLustr Tunnel (BHS64 Caddy → WireGuard wg1 10.67.0.12).16Everything goes through the gateway M1M32 and `mld` (`~/Desktop/cluster-skill/mld`).1718```19Internet → GoDaddy DNS (A www.market-atlas.co → 51.161.112.61)20        → BHS64 Caddy (TLS, WebSocket passthrough) → wg1 → M2U64:8380  market-atlas-core (Fastify edge: /v1 REST + WS + SSE, workers, connectors)21                                                                    └─ proxies every non-/v1 path → 127.0.0.1:8382  market-atlas-web (Next.js)22                                                                    └─ Postgres 17 Homebrew (db/role market_atlas) · ~/market-atlas-data (raw, archive, backups, logs)23```2425Node prerequisites (present on M2U64 on 2026-09-12): node 25, pnpm 9, pm2, `postgresql@17` (brew service), role/db `market_atlas`26(password `market_atlas`, localhost only). Data dirs: `~/market-atlas-data/{raw,archive,backups,logs,cache}`.2728## Release procedure (from the laptop)2930```bash31cd ~/Desktop/Projets/apps-web/market-atlas32pnpm typecheck && pnpm test                                          # 1. types + 58+ unit/connector/consensus/calendar tests33deploy/render-manifest.sh --push                                     # 2. manifest with MA_ADMIN_TOKEN + HFMD_API_KEY → M1M32:~/dispatch/apps/market-atlas.json34~/Desktop/cluster-skill/mld stage ~/Desktop/Projets/apps-web/market-atlas market-atlas   # 3. laptop → gateway staging (sync_excludes)35~/Desktop/cluster-skill/mld deploy market-atlas --node M2U64        # 4. rsync → node, hooks (pnpm install, esbuild, migrate+seed, next build), PM2, health, tunnel route, registry36curl -s https://www.market-atlas.co/v1/health                        # 5. public check37```3839The `post_sync` hooks build the API bundle (`apps/api/dist`), run migrations + seeds and build the web app. `main.js` also migrates and40seeds at start (idempotent). `mld heal` (every 5 min on M1M32) restarts anything missing after a power cut and keeps the Caddy route41pointing at the current node.4243## Operations4445```bash46~/Desktop/cluster-skill/mld status | grep market-atlas47ssh M2U64 'pm2 logs market-atlas-core --lines 100 --nostream'48ssh M2U64 'cd ~/apps/market-atlas && export PATH=/opt/homebrew/opt/postgresql@17/bin:/opt/homebrew/bin:$PATH DATABASE_URL=postgres://market_atlas:market_atlas@127.0.0.1:5432/market_atlas MA_DATA_DIR=$HOME/market-atlas-data && node apps/api/dist/cli.js status'49ssh M2U64 '… node apps/api/dist/cli.js run cboe-delayed-quotes'      # one poll, printed50ssh M2U64 '… node apps/api/dist/cli.js replay 2026-09-12'            # re-normalize archived raw payloads offline51ssh M2U64 '… node apps/api/dist/cli.js backup'                       # pg_dump of metadata → ~/market-atlas-data/backups (also nightly ~04:00)52curl -s https://www.market-atlas.co/v1/stats | python3 -m json.tool  # live counters53```5455Admin console: `https://www.market-atlas.co/admin` (token = `deploy/.admin-token`). Connector actions (pause/resume/restart/test),56schema-change acknowledgements, storage/partitions, discovery form. Environment variables are listed in `.env.example`.5758Retention: observations partitions older than `MA_OBSERVATION_RETENTION_DAYS` (45) are exported to `~/market-atlas-data/archive/<year>/…ndjson.gz`59then dropped (index in `observation_archives`). Bars, events, filings, quotes are permanent. Off-node copies of `backups/` and `archive/`60should be scheduled to M1M32 (`rsync -a M2U64:~/market-atlas-data/backups/ M1M32:~/backups/market-atlas/`).6162Apex domain: add `A market-atlas.co → 51.161.112.61` at GoDaddy, then set `"redirects": ["market-atlas.co"]` in the manifest and redeploy.63