spb/social-runtime-crawler

1# Social Runtime Crawler23Browser-native, self-learning crawler for modern social platforms. Instead of downloading documents, it **observes the runtime of a social application** through an authenticated browser (Playwright + Chromium), fuses several observation surfaces (network, DOM, media, navigation), builds a world model, lets an information-gain agent decide what to look at next, and progressively **learns how the platform works** (response schemas, page types, action → observation patterns) into a compiled connector.45Full specification: [`CLAUDE.md`](./CLAUDE.md). Architecture notes: [`docs/ARCHITECTURE.md`](./docs/ARCHITECTURE.md).67## Status — v0.1 proof of concept (2026-09-11)89Implemented (Phases 0 → 8 of §75, plus media detection and platform learning):1011| Area | What works |12|---|---|13| Browser runtime | persistent Chromium profile per (platform, account), human-in-the-loop login, CDP session |14| Network surface | XHR/fetch/GraphQL/WebSocket/media capture, `SchemaProfiler` (field semantics without property-name assumptions), generic JSON entity miner, endpoint fingerprints |15| DOM surface | injected `MutationObserver` deltas by region, in-page semantic snapshot (links, articles, videos, search, expand controls) |16| Page classification | URL hints + landmarks + density, confidence, `UNKNOWN` never crashes |17| Entities | URL-grammar extraction, network ↔ DOM merge with per-field provenance, agreements/conflicts report, identity-resolution skeleton |18| Agent | semantic action vocabulary, information-gain scoring, lexical novelty, loop detector, heuristic planner + optional LLM planner (Anthropic or local OpenAI-compatible) |19| Media | video detection (DOM `<video>` + manifests/segments + video entities), frame sampling at LEVEL ≥ 2 |20| Learning | `data/platform_model/<platform>/` (response, action, page, media patterns), degradation signal, connector manifest compilation |21| Storage | PostgreSQL (raw observations, entities, media, actions, schemas, feed items, relationships) + JSONL replay log |22| Console | **www.socialcrawl.co** — Next 16 research console (`apps/dashboard`): observatory, session console (decisions, page state, entities, runtime APIs, media, world-model graph, live events), entity evidence, runtime API explorer, connectors, mission launcher. Passcode gate. |23| Control plane | `apps/api` (node:http, port 8350): overview/sessions/entities/schemas/platforms queries, SSE `/api/stream`, job runner (spawns worker processes, `crawl_jobs`). Mutations need `SRC_API_TOKEN`. |24| Adapters | YouTube, Reddit (Phase 1), Facebook (Phase 2 — §52 prototype validated 2026-09-12 with an authenticated profile: search → group → public page → posts → profiles, GraphQL operations learned automatically). Instagram/TikTok/X/LinkedIn/Threads: not yet |2526Read-only by construction: no like/follow/comment/share/subscribe code path exists.2728Runtime data (`data/`: browser profiles, sessions, media) is excluded from git **and** from `mld stage` through `.mldignore` — an authenticated29profile must never leave the machine it was created on. On the laptop, `SRC_DATA_DIR` can point outside the repository.3031## Quick start3233```bash34pnpm install35cp .env.example .env            # set SRC_DATABASE_URL (createdb social_runtime) — optional36pnpm db:migrate                 # creates the tables3738pnpm login youtube              # opens Chromium, log in by hand, press Enter → profile saved39pnpm crawl youtube --query "intelligence artificielle Québec" --goal "Discover public Quebec creators discussing AI" --minutes 1040SRC_API_PORT=8350 pnpm api      # control-plane API41pnpm dashboard:dev              # console on http://localhost:8351 (API_URL=http://127.0.0.1:8350; set SRC_DASHBOARD_PASSCODE to enable the gate)4243pnpm learn youtube --minutes 15 # PLATFORM LEARNING mode + connector compilation → connectors/youtube/manifest.json44pnpm src status youtube         # "Platform learned: page types / entity types / schemas / confidence"45pnpm replay                     # list sessions ; pnpm replay <session_id> [--type ACTION_PLANNED]46pnpm test && pnpm typecheck47```4849Flags for `crawl`: `--mode research|observe|topic|profile|learn`, `--seed <url>`, `--minutes`, `--actions`, `--profiles`, `--posts`, `--videos`, `--media 0-4` (capture level, default 1), `--headless`, `--account <alias>`.5051LLM planner: set `SRC_LLM_PROVIDER=anthropic` (uses `ANTHROPIC_API_KEY` or an `ant auth login` profile, model `claude-opus-5`) or `SRC_LLM_PROVIDER=local` with `SRC_LOCAL_LLM_URL` / `SRC_LOCAL_LLM_MODEL` (e.g. llm-api.io on the cluster). The LLM is only consulted when the heuristic is unsure, only sees the compact page state, and can only pick an existing action id.5253## Deployment (MacLustr)5455`deploy/social-runtime-crawler.mld.json` → `M1M32:~/dispatch/apps/social-runtime-crawler.json`. `mld stage . social-runtime-crawler && mld deploy social-runtime-crawler`56places the app on the best node (Postgres 17 local `social_runtime`, PM2 `socialcrawl-api` :8350 + `socialcrawl-web` :8351, Chromium installed by hook) and routes57`https://www.socialcrawl.co` through the MacLustr Tunnel (BHS64). Browser profiles live on the node (`~/apps/social-runtime-crawler/data/browser_profiles`): log in once58through Screen Sharing with `pnpm login <platform>`. Secrets (API token, passcode) live only in the manifest on the gateway.5960## Layout6162```63apps/worker      CLI + crawl engine (observe → plan → act → learn loop) + action executor64apps/api         control-plane API (node:http): queries, SSE stream, job runner65apps/dashboard   SocialCrawl research console (Next 16 + React 19 + Tailwind 4 + recharts + d3-force)66packages/shared  types (platforms, surfaces, entities, actions, budgets), utils, logger, config67packages/events  Universal Social Event Format, typed bus, JSONL replay log68packages/browser SocialBrowserSession (persistent profile), interactive login69packages/observers  network (NetworkObserver, ResponseClassifier, SchemaProfiler), dom (DomObserver, PageSummarizer), PageClassifier70packages/entities   surface merge with provenance, confidence, IdentityResolver71packages/media      video detection, frame sampler72packages/agent      WorldModel, InformationGain, LoopDetector, planners, LLM clients73packages/platform-model  learned platform knowledge + connector compiler74packages/storage    PostgreSQL store + schema.sql75packages/connectors adapter interface, generic DOM→entity extraction, YouTube + Reddit adapters76connectors/<platform>/  compiled (learned) manifests — generated, not hand-written truth77data/            browser_profiles/ (never commit), sessions/<id>/events.jsonl, platform_model/, media/78```7980## Boundaries (§4)8182Only information the authenticated account can see through normal use. No bypass of authentication, access controls, CAPTCHA, rate limits, private content or privacy settings. Credentials are never typed or stored by the crawler; the browser profile is the only persisted authentication state and lives outside git.83