SPB Git forge
spboucher.ai ↗

spb/websensor

Public
33commits 1branches 0releases
3.4 MBsize
maindefault branch
10 days agolast push
TypeScript 55.4% Python 43.2% SQL 1.2%
7.3 KB
View source Raw Blame History

# Deploying WebSensor on MacLustr

Since 2026-09-13 production runs on the OVH server BHS128 (ssh BHS128, ubuntu@51.161.112.69, Ubuntu 24.04, 12 threads / 128 GB), outside mld: code in ~/apps/websensor, Postgres 17 native (db/role websensor/websensor, C.UTF-8 like the Mac source — with en_US.UTF-8 the generated events.search tsvector rejected a 5 836-byte keyword), Redis native, blobs in ~/websensor-data/blobs, PM2 under systemd (pm2-ubuntu, LimitNOFILE 1 048 576 for the 48 fetch workers) with ~/apps/pm2.websensor.config.cjs generated from the mld manifest by ~/apps/.manifests/gen-pm2.py (secrets, 0600). Tunnel route: mlt add www.websensor.io BHS128:8260 (BHS64 Caddy → wg1 10.67.0.60); firewall ufw allow in on wg1 from 10.67.0.1 to any port 8260 proto tcp. The move was triggered by the LAN uplink (Bell) collapsing under the ~6 000 outbound connections of the engine + factory (NAT table). Release on BHS128: rsync -az --exclude node_modules --exclude .git --exclude 'apps/web/.next' --exclude data --exclude 'deploy/*.mld.json' . BHS128:apps/websensor/ then ssh BHS128 'cd apps/websensor && pnpm install --frozen-lockfile && DATABASE_URL=postgres://websensor:websensor@127.0.0.1:5432/websensor pnpm db:migrate && DATABASE_URL=postgres://websensor:websensor@127.0.0.1:5432/websensor node node_modules/tsx/dist/cli.mjs apps/engine/src/cli.ts sync && API_URL=http://127.0.0.1:8260 NEXT_PUBLIC_SITE_URL=https://www.websensor.io pnpm --filter @websensor/web build && pm2 restart websensor-engine websensor-factory websensor-api websensor-web && pm2 save'. The M4M64b copy (code, blobs, Postgres database, Redis keys) was deleted on 2026-09-14: BHS128 now holds the only production data — schedule an off-host backup (pg_dump websensor + ~/websensor-data/blobs). The mld procedure below is kept for reference (manifest backup: laptop ~/Desktop/Cluster/secrets/mld-manifests-retired-20260914/websensor.json).

WebSensor formerly ran as four PM2 processes on one node (M4M64b), published through the MacLustr Tunnel (WireGuard + Caddy on the OVH gateway BHS64, www.websensor.io; ngrok was retired on 2026-09-10), orchestrated by mld from the M1M32 gateway.

text
Internet → www.websensor.io → BHS64 (Caddy TLS) → wg1 → websensor-api (:8260, Fastify)
                                         ├── /api/v1/*  REST + WebSocket /api/v1/live
                                         └── /*         reverse proxy → websensor-web (Next.js :8261, loopback)
websensor-engine (scheduler + pipeline, Prometheus :8262) → PostgreSQL 17 `websensor` + Redis (loopback)
websensor-factory (Source Factory: coverage seeds → deep discovery → shadow sensors → acceptance; heartbeat Redis ws:factory:status)
Blob store: ~/websensor-data/blobs (content-addressed, zstd) — never synced by mld (excluded `data/`).

# First deployment

  1. Copy deploy/websensor.mld.json.example to M1M32:~/dispatch/apps/websensor.json and fill the secrets (ANTHROPIC_API_KEY, SCRAPFLY_API_KEY, and WS_ADMIN_TOKEN on the API process — it enables /api/v1/admin/* and the /ops dashboard; a copy of the token lives in ~/.websensor/admin-token on the laptop). The manifest is 0600 on the gateway; a gitignored local copy may live in deploy/websensor.mld.json.
  2. ~/Desktop/cluster-skill/mld stage ~/Desktop/Projets/apps-web/websensor websensor
  3. ~/Desktop/cluster-skill/mld deploy websensor --node M4M64b Post-sync hooks: pnpm install, createdb + migrations, registry sync, next build. Health: /api/ready (database + engine activity in the last 15 minutes), public https://www.websensor.io/api/ready.
  4. Verify: mld status --live, curl https://www.websensor.io/api/v1/stats, and a WebSocket handshake (npx wscat -c wss://www.websensor.io/api/v1/live).

# DNS / tunnel

  • www.websensor.io → A record to BHS64 (51.161.112.61); mld deploy/move repoints the Caddy route to the current node automatically (tunnel.domain in the manifest). The apex websensor.io should redirect to www.
  • WebSocket upgrades traverse Caddy without configuration; the API sends a heartbeat every 25 s and clients replay missed events with {"since": lastSid} after a reconnection.

# Source Factory (0.3)

  • Seeds: config/coverage/*.yaml (universes) and config/factory/seeds/*.yaml; the post-sync hook runs cli.ts factory seed --mode=hinted (members without a monitored domain + members with hints). --mode=all --requeue re-discovers everything; factory requeue re-queues blocked/errored seeds.
  • Process: pm2 logs websensor-factory; cli.ts factory stats; /ops → Source Factory panel (funnel, sectors, candidates, manual accept/reject); GET /api/v1/admin/factory, /admin/factory/seeds, /admin/factory/candidates, POST /admin/factory/seeds (bulk), POST /admin/factory/shadow/:sensorId/accept|reject.
  • Tuning (env on the factory process): WS_FACTORY_CONCURRENCY (6), WS_FACTORY_BUDGET (90 requests/org), WS_FACTORY_MIN_SCORE (0.35), WS_FACTORY_SHADOW_MIN_CHECKS (5), WS_FACTORY_SHADOW_MIN_HOURS (24), WS_FACTORY_SHADOW_MAX_HOURS (120). GITHUB_TOKEN (optional) improves repository ranking.
  • Shadow sensors are polled by the engine at priority 3; raise WS_FETCH_CONCURRENCY on the engine when the shadow population grows (32 at 0.3). cli.ts factory evaluate [--force] runs the decision pass by hand; cli.ts factory export --sector=x > config/sources.d/6x-factory-x.yaml graduates accepted sensors into the registry.
  • Coverage: GET /api/v1/coverage, /coverage page; universes validated with apps/engine/src/coverage-validate.ts --dns.

# Operations

  • Logs: mld logs websensor or ssh M4M64b pm2 logs websensor-engine.
  • Registry edits: change config/sources.yaml, redeploy (sync runs in post-sync) or ssh M4M64b "cd ~/apps/websensor && node node_modules/tsx/dist/cli.mjs apps/engine/src/cli.ts sync". Seed sensors removed from the YAML are disabled automatically; discovery-created sensors are kept.
  • Discovery: cli.ts discover [sourceId…]; cli.ts probe <domain> for a dry run.
  • Metrics: engine http://127.0.0.1:8262/metrics, API /api/metrics (Prometheus text). Public health dashboard: /health on the site, /api/v1/health/connectors.
  • Budgets: WS_LLM_DAILY_CALL_BUDGET (Claude calls/day), WS_SCRAPFLY_DAILY_BUDGET (Scrapfly calls/day, ~30 credits each with anti-bot). Both counters reset at UTC midnight and are visible in metrics_daily.
  • Backups: pg_dump websensor + ~/websensor-data/blobs (content-addressed; safe to rsync incrementally).
  • Retention: sensor_runs unchanged/304 rows are pruned after 14 days, all runs after 60 days. Raw bodies of snapshots that never produced an event are pruned after WS_RETENTION_RAW_DAYS (21) — canonical forms, hashes, changes and events are kept; event snapshots are never pruned. cli.ts prune-blobs runs it by hand.
  • After a deploy that changes alias or scoring rules: cli.ts relink-entities 7 (re-resolves mentioned entities) and cli.ts refresh-clusters (breaking / developing / confirmed states).
  • Operations dashboard: /ops (token) and GET /api/v1/admin/ops; connector dry-run POST /api/v1/admin/sensors/test; bulk import POST /api/v1/admin/sources/import?dry_run=1 (JSON or YAML) — imported sensors start as PENDING and become ACTIVE after their first successful check.