TypeScript 55.4%
Python 43.2%
SQL 1.2%
1# Deploying WebSensor on MacLustr23> **Since 2026-09-13 production runs on the OVH server BHS128** (`ssh BHS128`, ubuntu@51.161.112.69, Ubuntu 24.04, 12 threads / 128 GB),4> outside `mld`: code in `~/apps/websensor`, Postgres 17 native (db/role `websensor`/`websensor`, **C.UTF-8** like the Mac source — with5> en_US.UTF-8 the generated `events.search` tsvector rejected a 5 836-byte keyword), Redis native, blobs in `~/websensor-data/blobs`,6> PM2 under systemd (`pm2-ubuntu`, `LimitNOFILE` 1 048 576 for the 48 fetch workers) with `~/apps/pm2.websensor.config.cjs` generated7> from the mld manifest by `~/apps/.manifests/gen-pm2.py` (secrets, 0600). Tunnel route: `mlt add www.websensor.io BHS128:8260`8> (BHS64 Caddy → wg1 10.67.0.60); firewall `ufw allow in on wg1 from 10.67.0.1 to any port 8260 proto tcp`. The move was triggered by the9> LAN uplink (Bell) collapsing under the ~6 000 outbound connections of the engine + factory (NAT table).10> **Release on BHS128**: `rsync -az --exclude node_modules --exclude .git --exclude 'apps/web/.next' --exclude data --exclude 'deploy/*.mld.json' . BHS128:apps/websensor/`11> then `ssh BHS128 'cd apps/websensor && pnpm install --frozen-lockfile && DATABASE_URL=postgres://websensor:websensor@127.0.0.1:5432/websensor pnpm db:migrate && DATABASE_URL=postgres://websensor:websensor@127.0.0.1:5432/websensor node node_modules/tsx/dist/cli.mjs apps/engine/src/cli.ts sync && API_URL=http://127.0.0.1:8260 NEXT_PUBLIC_SITE_URL=https://www.websensor.io pnpm --filter @websensor/web build && pm2 restart websensor-engine websensor-factory websensor-api websensor-web && pm2 save'`.12> The M4M64b copy (code, blobs, Postgres database, Redis keys) was deleted on 2026-09-14: BHS128 now holds the only production data —13> schedule an off-host backup (`pg_dump websensor` + `~/websensor-data/blobs`). The `mld` procedure below is kept for reference (manifest backup:14> laptop `~/Desktop/Cluster/secrets/mld-manifests-retired-20260914/websensor.json`).1516WebSensor formerly ran as four PM2 processes on one node (M4M64b), published through the MacLustr Tunnel (WireGuard +17Caddy on the OVH gateway BHS64, `www.websensor.io`; ngrok was retired on 2026-09-10), orchestrated by `mld` from the18M1M32 gateway.1920```21Internet → www.websensor.io → BHS64 (Caddy TLS) → wg1 → websensor-api (:8260, Fastify)22 ├── /api/v1/* REST + WebSocket /api/v1/live23 └── /* reverse proxy → websensor-web (Next.js :8261, loopback)24websensor-engine (scheduler + pipeline, Prometheus :8262) → PostgreSQL 17 `websensor` + Redis (loopback)25websensor-factory (Source Factory: coverage seeds → deep discovery → shadow sensors → acceptance; heartbeat Redis ws:factory:status)26Blob store: ~/websensor-data/blobs (content-addressed, zstd) — never synced by mld (excluded `data/`).27```2829## First deployment301. Copy `deploy/websensor.mld.json.example` to `M1M32:~/dispatch/apps/websensor.json` and fill the secrets31 (`ANTHROPIC_API_KEY`, `SCRAPFLY_API_KEY`, and `WS_ADMIN_TOKEN` on the API process — it enables `/api/v1/admin/*`32 and the `/ops` dashboard; a copy of the token lives in `~/.websensor/admin-token` on the laptop). The manifest is `0600` on the gateway; a gitignored local copy33 may live in `deploy/websensor.mld.json`.342. `~/Desktop/cluster-skill/mld stage ~/Desktop/Projets/apps-web/websensor websensor`353. `~/Desktop/cluster-skill/mld deploy websensor --node M4M64b`36 Post-sync hooks: `pnpm install`, `createdb` + migrations, registry sync, `next build`. Health: `/api/ready`37 (database + engine activity in the last 15 minutes), public `https://www.websensor.io/api/ready`.384. Verify: `mld status --live`, `curl https://www.websensor.io/api/v1/stats`, and a WebSocket handshake39 (`npx wscat -c wss://www.websensor.io/api/v1/live`).4041## DNS / tunnel42- `www.websensor.io` → A record to BHS64 (51.161.112.61); `mld deploy`/`move` repoints the Caddy route to the43 current node automatically (`tunnel.domain` in the manifest). The apex `websensor.io` should redirect to `www`.44- WebSocket upgrades traverse Caddy without configuration; the API sends a heartbeat every 25 s and clients45 replay missed events with `{"since": lastSid}` after a reconnection.4647## Source Factory (0.3)48- Seeds: `config/coverage/*.yaml` (universes) and `config/factory/seeds/*.yaml`; the post-sync hook runs49 `cli.ts factory seed --mode=hinted` (members without a monitored domain + members with hints). `--mode=all --requeue`50 re-discovers everything; `factory requeue` re-queues blocked/errored seeds.51- Process: `pm2 logs websensor-factory`; `cli.ts factory stats`; `/ops` → Source Factory panel (funnel, sectors,52 candidates, manual accept/reject); `GET /api/v1/admin/factory`, `/admin/factory/seeds`, `/admin/factory/candidates`,53 `POST /admin/factory/seeds` (bulk), `POST /admin/factory/shadow/:sensorId/accept|reject`.54- Tuning (env on the factory process): `WS_FACTORY_CONCURRENCY` (6), `WS_FACTORY_BUDGET` (90 requests/org),55 `WS_FACTORY_MIN_SCORE` (0.35), `WS_FACTORY_SHADOW_MIN_CHECKS` (5), `WS_FACTORY_SHADOW_MIN_HOURS` (24),56 `WS_FACTORY_SHADOW_MAX_HOURS` (120). `GITHUB_TOKEN` (optional) improves repository ranking.57- Shadow sensors are polled by the engine at priority 3; raise `WS_FETCH_CONCURRENCY` on the engine when the shadow58 population grows (32 at 0.3). `cli.ts factory evaluate [--force]` runs the decision pass by hand;59 `cli.ts factory export --sector=x > config/sources.d/6x-factory-x.yaml` graduates accepted sensors into the registry.60- Coverage: `GET /api/v1/coverage`, `/coverage` page; universes validated with `apps/engine/src/coverage-validate.ts --dns`.6162## Operations63- Logs: `mld logs websensor` or `ssh M4M64b pm2 logs websensor-engine`.64- Registry edits: change `config/sources.yaml`, redeploy (sync runs in post-sync) or65 `ssh M4M64b "cd ~/apps/websensor && node node_modules/tsx/dist/cli.mjs apps/engine/src/cli.ts sync"`.66 Seed sensors removed from the YAML are disabled automatically; discovery-created sensors are kept.67- Discovery: `cli.ts discover [sourceId…]`; `cli.ts probe <domain>` for a dry run.68- Metrics: engine `http://127.0.0.1:8262/metrics`, API `/api/metrics` (Prometheus text). Public health dashboard:69 `/health` on the site, `/api/v1/health/connectors`.70- Budgets: `WS_LLM_DAILY_CALL_BUDGET` (Claude calls/day), `WS_SCRAPFLY_DAILY_BUDGET` (Scrapfly calls/day, ~3071 credits each with anti-bot). Both counters reset at UTC midnight and are visible in `metrics_daily`.72- Backups: `pg_dump websensor` + `~/websensor-data/blobs` (content-addressed; safe to rsync incrementally).73- Retention: `sensor_runs` unchanged/304 rows are pruned after 14 days, all runs after 60 days. Raw bodies of74 snapshots that never produced an event are pruned after `WS_RETENTION_RAW_DAYS` (21) — canonical forms, hashes,75 changes and events are kept; event snapshots are never pruned. `cli.ts prune-blobs` runs it by hand.76- After a deploy that changes alias or scoring rules: `cli.ts relink-entities 7` (re-resolves mentioned entities)77 and `cli.ts refresh-clusters` (breaking / developing / confirmed states).78- Operations dashboard: `/ops` (token) and `GET /api/v1/admin/ops`; connector dry-run `POST /api/v1/admin/sensors/test`;79 bulk import `POST /api/v1/admin/sources/import?dry_run=1` (JSON or YAML) — imported sensors start as PENDING and80 become ACTIVE after their first successful check.81