README: console, control plane, MacLustr deployment; QA scripts

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Simon-Pierre Boucher committed 12 days ago (Sep 12, 2026) parent d16c8f9

2 changed files +26 −3

modified README.md +13 −3

		`@@ -19,7 +19,8 @@ Implemented (Phases 0 → 8 of §75, plus media detection and platform learning)`
19	19	\| Media \| video detection (DOM `<video>` + manifests/segments + video entities), frame sampling at LEVEL ≥ 2 \|
20	20	\| Learning \| `data/platform_model/<platform>/` (response, action, page, media patterns), degradation signal, connector manifest compilation \|
21	21	\| Storage \| PostgreSQL (raw observations, entities, media, actions, schemas, feed items, relationships) + JSONL replay log \|
22		−\| Dashboard \| `pnpm dashboard` → http://localhost:8340 (decisions, page state, entities, schemas, media, connector confidence, world model, raw events) \|
	22	+\| Console \| www.socialcrawl.co — Next 16 research console (`apps/dashboard`): observatory, session console (decisions, page state, entities, runtime APIs, media, world-model graph, live events), entity evidence, runtime API explorer, connectors, mission launcher. Passcode gate. \|
	23	+\| Control plane \| `apps/api` (node:http, port 8350): overview/sessions/entities/schemas/platforms queries, SSE `/api/stream`, job runner (spawns worker processes, `crawl_jobs`). Mutations need `SRC_API_TOKEN`. \|
23	24	\| Adapters \| YouTube, Reddit (Phase 1). Facebook/Instagram/TikTok/X/LinkedIn/Threads: not yet \|
24	25
25	26	Read-only by construction: no like/follow/comment/share/subscribe code path exists.
		`@@ -33,7 +34,8 @@ pnpm db:migrate # creates the tables`
33	34
34	35	pnpm login youtube # opens Chromium, log in by hand, press Enter → profile saved
35	36	pnpm crawl youtube --query "intelligence artificielle Québec" --goal "Discover public Quebec creators discussing AI" --minutes 10
36		−pnpm dashboard # research console
	37	+SRC_API_PORT=8350 pnpm api # control-plane API
	38	+pnpm dashboard:dev # console on http://localhost:8351 (API_URL=http://127.0.0.1:8350; set SRC_DASHBOARD_PASSCODE to enable the gate)
37	39
38	40	pnpm learn youtube --minutes 15 # PLATFORM LEARNING mode + connector compilation → connectors/youtube/manifest.json
39	41	pnpm src status youtube # "Platform learned: page types / entity types / schemas / confidence"
		@@ -45,11 +47,19 @@ Flags for `crawl`: `--mode research\|observe\|topic\|profile\|learn`, `--seed <url>`
45	47
46	48	LLM planner: set `SRC_LLM_PROVIDER=anthropic` (uses `ANTHROPIC_API_KEY` or an `ant auth login` profile, model `claude-opus-5`) or `SRC_LLM_PROVIDER=local` with `SRC_LOCAL_LLM_URL` / `SRC_LOCAL_LLM_MODEL` (e.g. llm-api.io on the cluster). The LLM is only consulted when the heuristic is unsure, only sees the compact page state, and can only pick an existing action id.
47	49
	50	+## Deployment (MacLustr)
	51	+
	52	+`deploy/social-runtime-crawler.mld.json` → `M1M32:~/dispatch/apps/social-runtime-crawler.json`. `mld stage . social-runtime-crawler && mld deploy social-runtime-crawler`
	53	+places the app on the best node (Postgres 17 local `social_runtime`, PM2 `socialcrawl-api` :8350 + `socialcrawl-web` :8351, Chromium installed by hook) and routes
	54	+`https://www.socialcrawl.co` through the MacLustr Tunnel (BHS64). Browser profiles live on the node (`~/apps/social-runtime-crawler/data/browser_profiles`): log in once
	55	+through Screen Sharing with `pnpm login <platform>`. Secrets (API token, passcode) live only in the manifest on the gateway.
	56	+
48	57	## Layout
49	58
50	59	```
51	60	apps/worker CLI + crawl engine (observe → plan → act → learn loop) + action executor
52		−apps/api dashboard server (node:http) + public/index.html
	61	+apps/api control-plane API (node:http): queries, SSE stream, job runner
	62	+apps/dashboard SocialCrawl research console (Next 16 + React 19 + Tailwind 4 + recharts + d3-force)
53	63	packages/shared types (platforms, surfaces, entities, actions, budgets), utils, logger, config
54	64	packages/events Universal Social Event Format, typed bus, JSONL replay log
55	65	packages/browser SocialBrowserSession (persistent profile), interactive login

added apps/worker/qa/public-e2e.mjs +13 −0

		`@@ -0,0 +1,13 @@`
	1	+import { chromium } from "playwright";
	2	+const BASE = process.env.BASE ?? "https://www.socialcrawl.co";
	3	+const CODE = process.env.CODE;
	4	+const b = await chromium.launch(); const p = await b.newPage({ viewport: { width: 1500, height: 1000 } });
	5	+await p.goto(`${BASE}/access`); await p.fill('input[name=passcode]', CODE); await p.click("button"); await p.waitForURL(`${BASE}/`);
	6	+const job = await p.evaluate(async () => { const r = await fetch("/api/jobs", { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ platform: "youtube", mode: "topic", query: "intelligence artificielle Québec", goal: "Discover public Quebec creators and channels discussing artificial intelligence", minutes: 2, actions: 8, media: 1, headless: true }) }); return { status: r.status, body: await r.text() }; });
	7	+console.log("launch:", JSON.stringify(job));
	8	+await p.goto(`${BASE}/jobs`); await p.waitForTimeout(25000); await p.screenshot({ path: "/tmp/pub-jobs.png" });
	9	+await p.goto(`${BASE}/`); await p.waitForTimeout(12000); await p.screenshot({ path: "/tmp/pub-home.png", fullPage: false });
	10	+const sessions = await p.evaluate(async () => (await (await fetch("/api/sessions?limit=1")).json()));
	11	+const sid = sessions[0]?.session_id; console.log("session:", sid, sessions[0]?.goal);
	12	+if (sid) { await p.goto(`${BASE}/sessions/${sid}`); await p.waitForTimeout(6000); await p.screenshot({ path: "/tmp/pub-session.png", fullPage: true }); await p.click("text=events"); await p.waitForTimeout(6000); await p.screenshot({ path: "/tmp/pub-events.png" }); }
	13	+await b.close();
	14