SPB Git
spboucher.ai

Repository Public

spbsocial-runtime-crawler

Commit

README: console, control plane, MacLustr deployment; QA scripts

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed · Sep 12, 2026 parent d16c8f9
2 changed files +26 −3 Browse files
2 changed files
Unified Split
modified README.md +13 −3
@@ -19,7 +19,8 @@ Implemented (Phases 0 → 8 of §75, plus media detection and platform learning)
19 | Media | video detection (DOM `<video>` + manifests/segments + video entities), frame sampling at LEVEL ≥ 2 | 19 | Media | video detection (DOM `<video>` + manifests/segments + video entities), frame sampling at LEVEL ≥ 2 |
20 | Learning | `data/platform_model/<platform>/` (response, action, page, media patterns), degradation signal, connector manifest compilation | 20 | Learning | `data/platform_model/<platform>/` (response, action, page, media patterns), degradation signal, connector manifest compilation |
21 | Storage | PostgreSQL (raw observations, entities, media, actions, schemas, feed items, relationships) + JSONL replay log | 21 | Storage | PostgreSQL (raw observations, entities, media, actions, schemas, feed items, relationships) + JSONL replay log |
22−| Dashboard | `pnpm dashboard` → http://localhost:8340 (decisions, page state, entities, schemas, media, connector confidence, world model, raw events) | 22+| Console | **www.socialcrawl.co** — Next 16 research console (`apps/dashboard`): observatory, session console (decisions, page state, entities, runtime APIs, media, world-model graph, live events), entity evidence, runtime API explorer, connectors, mission launcher. Passcode gate. |
23+| Control plane | `apps/api` (node:http, port 8350): overview/sessions/entities/schemas/platforms queries, SSE `/api/stream`, job runner (spawns worker processes, `crawl_jobs`). Mutations need `SRC_API_TOKEN`. |
23 | Adapters | YouTube, Reddit (Phase 1). Facebook/Instagram/TikTok/X/LinkedIn/Threads: not yet | 24 | Adapters | YouTube, Reddit (Phase 1). Facebook/Instagram/TikTok/X/LinkedIn/Threads: not yet |
24 25
25 Read-only by construction: no like/follow/comment/share/subscribe code path exists. 26 Read-only by construction: no like/follow/comment/share/subscribe code path exists.
@@ -33,7 +34,8 @@ pnpm db:migrate # creates the tables
33 34
34 pnpm login youtube # opens Chromium, log in by hand, press Enter → profile saved 35 pnpm login youtube # opens Chromium, log in by hand, press Enter → profile saved
35 pnpm crawl youtube --query "intelligence artificielle Québec" --goal "Discover public Quebec creators discussing AI" --minutes 10 36 pnpm crawl youtube --query "intelligence artificielle Québec" --goal "Discover public Quebec creators discussing AI" --minutes 10
36−pnpm dashboard # research console 37+SRC_API_PORT=8350 pnpm api # control-plane API
38+pnpm dashboard:dev # console on http://localhost:8351 (API_URL=http://127.0.0.1:8350; set SRC_DASHBOARD_PASSCODE to enable the gate)
37 39
38 pnpm learn youtube --minutes 15 # PLATFORM LEARNING mode + connector compilation → connectors/youtube/manifest.json 40 pnpm learn youtube --minutes 15 # PLATFORM LEARNING mode + connector compilation → connectors/youtube/manifest.json
39 pnpm src status youtube # "Platform learned: page types / entity types / schemas / confidence" 41 pnpm src status youtube # "Platform learned: page types / entity types / schemas / confidence"
@@ -45,11 +47,19 @@ Flags for `crawl`: `--mode research|observe|topic|profile|learn`, `--seed <url>`
45 47
46 LLM planner: set `SRC_LLM_PROVIDER=anthropic` (uses `ANTHROPIC_API_KEY` or an `ant auth login` profile, model `claude-opus-5`) or `SRC_LLM_PROVIDER=local` with `SRC_LOCAL_LLM_URL` / `SRC_LOCAL_LLM_MODEL` (e.g. llm-api.io on the cluster). The LLM is only consulted when the heuristic is unsure, only sees the compact page state, and can only pick an existing action id. 48 LLM planner: set `SRC_LLM_PROVIDER=anthropic` (uses `ANTHROPIC_API_KEY` or an `ant auth login` profile, model `claude-opus-5`) or `SRC_LLM_PROVIDER=local` with `SRC_LOCAL_LLM_URL` / `SRC_LOCAL_LLM_MODEL` (e.g. llm-api.io on the cluster). The LLM is only consulted when the heuristic is unsure, only sees the compact page state, and can only pick an existing action id.
47 49
50+## Deployment (MacLustr)
51+
52+`deploy/social-runtime-crawler.mld.json` → `M1M32:~/dispatch/apps/social-runtime-crawler.json`. `mld stage . social-runtime-crawler && mld deploy social-runtime-crawler`
53+places the app on the best node (Postgres 17 local `social_runtime`, PM2 `socialcrawl-api` :8350 + `socialcrawl-web` :8351, Chromium installed by hook) and routes
54+`https://www.socialcrawl.co` through the MacLustr Tunnel (BHS64). Browser profiles live on the node (`~/apps/social-runtime-crawler/data/browser_profiles`): log in once
55+through Screen Sharing with `pnpm login <platform>`. Secrets (API token, passcode) live only in the manifest on the gateway.
56+
48 ## Layout 57 ## Layout
49 58
50 ``` 59 ```
51 apps/worker CLI + crawl engine (observe → plan → act → learn loop) + action executor 60 apps/worker CLI + crawl engine (observe → plan → act → learn loop) + action executor
52−apps/api dashboard server (node:http) + public/index.html 61+apps/api control-plane API (node:http): queries, SSE stream, job runner
62+apps/dashboard SocialCrawl research console (Next 16 + React 19 + Tailwind 4 + recharts + d3-force)
53 packages/shared types (platforms, surfaces, entities, actions, budgets), utils, logger, config 63 packages/shared types (platforms, surfaces, entities, actions, budgets), utils, logger, config
54 packages/events Universal Social Event Format, typed bus, JSONL replay log 64 packages/events Universal Social Event Format, typed bus, JSONL replay log
55 packages/browser SocialBrowserSession (persistent profile), interactive login 65 packages/browser SocialBrowserSession (persistent profile), interactive login
added apps/worker/qa/public-e2e.mjs +13 −0
@@ -0,0 +1,13 @@
1+import { chromium } from "playwright";
2+const BASE = process.env.BASE ?? "https://www.socialcrawl.co";
3+const CODE = process.env.CODE;
4+const b = await chromium.launch(); const p = await b.newPage({ viewport: { width: 1500, height: 1000 } });
5+await p.goto(`${BASE}/access`); await p.fill('input[name=passcode]', CODE); await p.click("button"); await p.waitForURL(`${BASE}/`);
6+const job = await p.evaluate(async () => { const r = await fetch("/api/jobs", { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ platform: "youtube", mode: "topic", query: "intelligence artificielle Québec", goal: "Discover public Quebec creators and channels discussing artificial intelligence", minutes: 2, actions: 8, media: 1, headless: true }) }); return { status: r.status, body: await r.text() }; });
7+console.log("launch:", JSON.stringify(job));
8+await p.goto(`${BASE}/jobs`); await p.waitForTimeout(25000); await p.screenshot({ path: "/tmp/pub-jobs.png" });
9+await p.goto(`${BASE}/`); await p.waitForTimeout(12000); await p.screenshot({ path: "/tmp/pub-home.png", fullPage: false });
10+const sessions = await p.evaluate(async () => (await (await fetch("/api/sessions?limit=1")).json()));
11+const sid = sessions[0]?.session_id; console.log("session:", sid, sessions[0]?.goal);
12+if (sid) { await p.goto(`${BASE}/sessions/${sid}`); await p.waitForTimeout(6000); await p.screenshot({ path: "/tmp/pub-session.png", fullPage: true }); await p.click("text=events"); await p.waitForTimeout(6000); await p.screenshot({ path: "/tmp/pub-events.png" }); }
13+await b.close();
14
Menu

Theme
↑↓navigate ↵open escclose /⌘Kanywhere