# Deploying WebSensor on MacLustr

WebSensor runs as four PM2 processes on one node (M4M64b), published through the MacLustr Tunnel (WireGuard + Caddy on the OVH gateway BHS64, www.websensor.io; ngrok was retired on 2026-09-10), orchestrated by mld from the M1M32 gateway.

Internet → www.websensor.io → BHS64 (Caddy TLS) → wg1 → websensor-api (:8260, Fastify)
                                         ├── /api/v1/*  REST + WebSocket /api/v1/live
                                         └── /*         reverse proxy → websensor-web (Next.js :8261, loopback)
websensor-engine (scheduler + pipeline, Prometheus :8262) → PostgreSQL 17 `websensor` + Redis (loopback)
websensor-factory (Source Factory: coverage seeds → deep discovery → shadow sensors → acceptance; heartbeat Redis ws:factory:status)
Blob store: ~/websensor-data/blobs (content-addressed, zstd) — never synced by mld (excluded `data/`).

# First deployment

1. Copy deploy/websensor.mld.json.example to M1M32:~/dispatch/apps/websensor.json and fill the secrets (ANTHROPIC_API_KEY, SCRAPFLY_API_KEY, and WS_ADMIN_TOKEN on the API process — it enables /api/v1/admin/* and the /ops dashboard; a copy of the token lives in ~/.websensor/admin-token on the laptop). The manifest is 0600 on the gateway; a gitignored local copy may live in deploy/websensor.mld.json.
2. ~/Desktop/cluster-skill/mld stage ~/Desktop/Projets/apps-web/websensor websensor
3. ~/Desktop/cluster-skill/mld deploy websensor --node M4M64b Post-sync hooks: pnpm install, createdb + migrations, registry sync, next build. Health: /api/ready (database + engine activity in the last 15 minutes), public https://www.websensor.io/api/ready.
4. Verify: mld status --live, curl https://www.websensor.io/api/v1/stats, and a WebSocket handshake (npx wscat -c wss://www.websensor.io/api/v1/live).

# DNS / tunnel

• www.websensor.io → A record to BHS64 (51.161.112.61); mld deploy/move repoints the Caddy route to the current node automatically (tunnel.domain in the manifest). The apex websensor.io should redirect to www.
• WebSocket upgrades traverse Caddy without configuration; the API sends a heartbeat every 25 s and clients replay missed events with {"since": lastSid} after a reconnection.

# Source Factory (0.3)

• Seeds: config/coverage/*.yaml (universes) and config/factory/seeds/*.yaml; the post-sync hook runs cli.ts factory seed --mode=hinted (members without a monitored domain + members with hints). --mode=all --requeue re-discovers everything; factory requeue re-queues blocked/errored seeds.
• Process: pm2 logs websensor-factory; cli.ts factory stats; /ops → Source Factory panel (funnel, sectors, candidates, manual accept/reject); GET /api/v1/admin/factory, /admin/factory/seeds, /admin/factory/candidates, POST /admin/factory/seeds (bulk), POST /admin/factory/shadow/:sensorId/accept|reject.
• Tuning (env on the factory process): WS_FACTORY_CONCURRENCY (6), WS_FACTORY_BUDGET (90 requests/org), WS_FACTORY_MIN_SCORE (0.35), WS_FACTORY_SHADOW_MIN_CHECKS (5), WS_FACTORY_SHADOW_MIN_HOURS (24), WS_FACTORY_SHADOW_MAX_HOURS (120). GITHUB_TOKEN (optional) improves repository ranking.
• Shadow sensors are polled by the engine at priority 3; raise WS_FETCH_CONCURRENCY on the engine when the shadow population grows (32 at 0.3). cli.ts factory evaluate [--force] runs the decision pass by hand; cli.ts factory export --sector=x > config/sources.d/6x-factory-x.yaml graduates accepted sensors into the registry.
• Coverage: GET /api/v1/coverage, /coverage page; universes validated with apps/engine/src/coverage-validate.ts --dns.

# Operations

• Logs: mld logs websensor or ssh M4M64b pm2 logs websensor-engine.
• Registry edits: change config/sources.yaml, redeploy (sync runs in post-sync) or ssh M4M64b "cd ~/apps/websensor && node node_modules/tsx/dist/cli.mjs apps/engine/src/cli.ts sync". Seed sensors removed from the YAML are disabled automatically; discovery-created sensors are kept.
• Discovery: cli.ts discover [sourceId…]; cli.ts probe <domain> for a dry run.
• Metrics: engine http://127.0.0.1:8262/metrics, API /api/metrics (Prometheus text). Public health dashboard: /health on the site, /api/v1/health/connectors.
• Budgets: WS_LLM_DAILY_CALL_BUDGET (Claude calls/day), WS_SCRAPFLY_DAILY_BUDGET (Scrapfly calls/day, ~30 credits each with anti-bot). Both counters reset at UTC midnight and are visible in metrics_daily.
• Backups: pg_dump websensor + ~/websensor-data/blobs (content-addressed; safe to rsync incrementally).
• Retention: sensor_runs unchanged/304 rows are pruned after 14 days, all runs after 60 days. Raw bodies of snapshots that never produced an event are pruned after WS_RETENTION_RAW_DAYS (21) — canonical forms, hashes, changes and events are kept; event snapshots are never pruned. cli.ts prune-blobs runs it by hand.
• After a deploy that changes alias or scoring rules: cli.ts relink-entities 7 (re-resolves mentioned entities) and cli.ts refresh-clusters (breaking / developing / confirmed states).
• Operations dashboard: /ops (token) and GET /api/v1/admin/ops; connector dry-run POST /api/v1/admin/sensors/test; bulk import POST /api/v1/admin/sources/import?dry_run=1 (JSON or YAML) — imported sensors start as PENDING and become ACTIVE after their first successful check.