1# Deploying WebSensor on MacLustr23WebSensor runs as four PM2 processes on one node (M4M64b), published through the MacLustr Tunnel (WireGuard +4Caddy on the OVH gateway BHS64, `www.websensor.io`; ngrok was retired on 2026-09-10), orchestrated by `mld` from the5M1M32 gateway.67```8Internet → www.websensor.io → BHS64 (Caddy TLS) → wg1 → websensor-api (:8260, Fastify)9 ├── /api/v1/* REST + WebSocket /api/v1/live10 └── /* reverse proxy → websensor-web (Next.js :8261, loopback)11websensor-engine (scheduler + pipeline, Prometheus :8262) → PostgreSQL 17 `websensor` + Redis (loopback)12websensor-factory (Source Factory: coverage seeds → deep discovery → shadow sensors → acceptance; heartbeat Redis ws:factory:status)13Blob store: ~/websensor-data/blobs (content-addressed, zstd) — never synced by mld (excluded `data/`).14```1516## First deployment171. Copy `deploy/websensor.mld.json.example` to `M1M32:~/dispatch/apps/websensor.json` and fill the secrets18 (`ANTHROPIC_API_KEY`, `SCRAPFLY_API_KEY`, and `WS_ADMIN_TOKEN` on the API process — it enables `/api/v1/admin/*`19 and the `/ops` dashboard; a copy of the token lives in `~/.websensor/admin-token` on the laptop). The manifest is `0600` on the gateway; a gitignored local copy20 may live in `deploy/websensor.mld.json`.212. `~/Desktop/cluster-skill/mld stage ~/Desktop/Projets/apps-web/websensor websensor`223. `~/Desktop/cluster-skill/mld deploy websensor --node M4M64b`23 Post-sync hooks: `pnpm install`, `createdb` + migrations, registry sync, `next build`. Health: `/api/ready`24 (database + engine activity in the last 15 minutes), public `https://www.websensor.io/api/ready`.254. Verify: `mld status --live`, `curl https://www.websensor.io/api/v1/stats`, and a WebSocket handshake26 (`npx wscat -c wss://www.websensor.io/api/v1/live`).2728## DNS / tunnel29- `www.websensor.io` → A record to BHS64 (51.161.112.61); `mld deploy`/`move` repoints the Caddy route to the30 current node automatically (`tunnel.domain` in the manifest). The apex `websensor.io` should redirect to `www`.31- WebSocket upgrades traverse Caddy without configuration; the API sends a heartbeat every 25 s and clients32 replay missed events with `{"since": lastSid}` after a reconnection.3334## Source Factory (0.3)35- Seeds: `config/coverage/*.yaml` (universes) and `config/factory/seeds/*.yaml`; the post-sync hook runs36 `cli.ts factory seed --mode=hinted` (members without a monitored domain + members with hints). `--mode=all --requeue`37 re-discovers everything; `factory requeue` re-queues blocked/errored seeds.38- Process: `pm2 logs websensor-factory`; `cli.ts factory stats`; `/ops` → Source Factory panel (funnel, sectors,39 candidates, manual accept/reject); `GET /api/v1/admin/factory`, `/admin/factory/seeds`, `/admin/factory/candidates`,40 `POST /admin/factory/seeds` (bulk), `POST /admin/factory/shadow/:sensorId/accept|reject`.41- Tuning (env on the factory process): `WS_FACTORY_CONCURRENCY` (6), `WS_FACTORY_BUDGET` (90 requests/org),42 `WS_FACTORY_MIN_SCORE` (0.35), `WS_FACTORY_SHADOW_MIN_CHECKS` (5), `WS_FACTORY_SHADOW_MIN_HOURS` (24),43 `WS_FACTORY_SHADOW_MAX_HOURS` (120). `GITHUB_TOKEN` (optional) improves repository ranking.44- Shadow sensors are polled by the engine at priority 3; raise `WS_FETCH_CONCURRENCY` on the engine when the shadow45 population grows (32 at 0.3). `cli.ts factory evaluate [--force]` runs the decision pass by hand;46 `cli.ts factory export --sector=x > config/sources.d/6x-factory-x.yaml` graduates accepted sensors into the registry.47- Coverage: `GET /api/v1/coverage`, `/coverage` page; universes validated with `apps/engine/src/coverage-validate.ts --dns`.4849## Operations50- Logs: `mld logs websensor` or `ssh M4M64b pm2 logs websensor-engine`.51- Registry edits: change `config/sources.yaml`, redeploy (sync runs in post-sync) or52 `ssh M4M64b "cd ~/apps/websensor && node node_modules/tsx/dist/cli.mjs apps/engine/src/cli.ts sync"`.53 Seed sensors removed from the YAML are disabled automatically; discovery-created sensors are kept.54- Discovery: `cli.ts discover [sourceId…]`; `cli.ts probe <domain>` for a dry run.55- Metrics: engine `http://127.0.0.1:8262/metrics`, API `/api/metrics` (Prometheus text). Public health dashboard:56 `/health` on the site, `/api/v1/health/connectors`.57- Budgets: `WS_LLM_DAILY_CALL_BUDGET` (Claude calls/day), `WS_SCRAPFLY_DAILY_BUDGET` (Scrapfly calls/day, ~3058 credits each with anti-bot). Both counters reset at UTC midnight and are visible in `metrics_daily`.59- Backups: `pg_dump websensor` + `~/websensor-data/blobs` (content-addressed; safe to rsync incrementally).60- Retention: `sensor_runs` unchanged/304 rows are pruned after 14 days, all runs after 60 days. Raw bodies of61 snapshots that never produced an event are pruned after `WS_RETENTION_RAW_DAYS` (21) — canonical forms, hashes,62 changes and events are kept; event snapshots are never pruned. `cli.ts prune-blobs` runs it by hand.63- After a deploy that changes alias or scoring rules: `cli.ts relink-entities 7` (re-resolves mentioned entities)64 and `cli.ts refresh-clusters` (breaking / developing / confirmed states).65- Operations dashboard: `/ops` (token) and `GET /api/v1/admin/ops`; connector dry-run `POST /api/v1/admin/sensors/test`;66 bulk import `POST /api/v1/admin/sources/import?dry_run=1` (JSON or YAML) — imported sensors start as PENDING and67 become ACTIVE after their first successful check.68