Social Runtime Crawler
Browser-native, self-learning crawler for modern social platforms. Instead of downloading documents, it observes the runtime of a social application through an authenticated browser (Playwright + Chromium), fuses several observation surfaces (network, DOM, media, navigation), builds a world model, lets an information-gain agent decide what to look at next, and progressively learns how the platform works (response schemas, page types, action → observation patterns) into a compiled connector.
Full specification: CLAUDE.md. Architecture notes: docs/ARCHITECTURE.md.
Status — v0.1 proof of concept (2026-09-11)
Implemented (Phases 0 → 8 of §75, plus media detection and platform learning):
| Area | What works |
|---|---|
| Browser runtime | persistent Chromium profile per (platform, account), human-in-the-loop login, CDP session |
| Network surface | XHR/fetch/GraphQL/WebSocket/media capture, SchemaProfiler (field semantics without property-name assumptions), generic JSON entity miner, endpoint fingerprints |
| DOM surface | injected MutationObserver deltas by region, in-page semantic snapshot (links, articles, videos, search, expand controls) |
| Page classification | URL hints + landmarks + density, confidence, UNKNOWN never crashes |
| Entities | URL-grammar extraction, network ↔ DOM merge with per-field provenance, agreements/conflicts report, identity-resolution skeleton |
| Agent | semantic action vocabulary, information-gain scoring, lexical novelty, loop detector, heuristic planner + optional LLM planner (Anthropic or local OpenAI-compatible) |
| Media | video detection (DOM <video> + manifests/segments + video entities), frame sampling at LEVEL ≥ 2 |
| Learning | data/platform_model/<platform>/ (response, action, page, media patterns), degradation signal, connector manifest compilation |
| Storage | PostgreSQL (raw observations, entities, media, actions, schemas, feed items, relationships) + JSONL replay log |
| Console | www.socialcrawl.co — Next 16 research console (apps/dashboard): observatory, session console (decisions, page state, entities, runtime APIs, media, world-model graph, live events), entity evidence, runtime API explorer, connectors, mission launcher. Passcode gate. |
| Control plane | apps/api (node:http, port 8350): overview/sessions/entities/schemas/platforms queries, SSE /api/stream, job runner (spawns worker processes, crawl_jobs). Mutations need SRC_API_TOKEN. |
| Adapters | YouTube, Reddit (Phase 1), Facebook (Phase 2 — §52 prototype validated 2026-09-12 with an authenticated profile: search → group → public page → posts → profiles, GraphQL operations learned automatically). Instagram/TikTok/X/LinkedIn/Threads: not yet |
Read-only by construction: no like/follow/comment/share/subscribe code path exists.
Runtime data (data/: browser profiles, sessions, media) is excluded from git and from mld stage through .mldignore — an authenticated
profile must never leave the machine it was created on. On the laptop, SRC_DATA_DIR can point outside the repository.
Quick start
pnpm install
cp .env.example .env # set SRC_DATABASE_URL (createdb social_runtime) — optional
pnpm db:migrate # creates the tables
pnpm login youtube # opens Chromium, log in by hand, press Enter → profile saved
pnpm crawl youtube --query "intelligence artificielle Québec" --goal "Discover public Quebec creators discussing AI" --minutes 10
SRC_API_PORT=8350 pnpm api # control-plane API
pnpm dashboard:dev # console on http://localhost:8351 (API_URL=http://127.0.0.1:8350; set SRC_DASHBOARD_PASSCODE to enable the gate)
pnpm learn youtube --minutes 15 # PLATFORM LEARNING mode + connector compilation → connectors/youtube/manifest.json
pnpm src status youtube # "Platform learned: page types / entity types / schemas / confidence"
pnpm replay # list sessions ; pnpm replay <session_id> [--type ACTION_PLANNED]
pnpm test && pnpm typecheckFlags for crawl: --mode research|observe|topic|profile|learn, --seed <url>, --minutes, --actions, --profiles, --posts, --videos, --media 0-4 (capture level, default 1), --headless, --account <alias>.
LLM planner: set SRC_LLM_PROVIDER=anthropic (uses ANTHROPIC_API_KEY or an ant auth login profile, model claude-opus-5) or SRC_LLM_PROVIDER=local with SRC_LOCAL_LLM_URL / SRC_LOCAL_LLM_MODEL (e.g. llm-api.io on the cluster). The LLM is only consulted when the heuristic is unsure, only sees the compact page state, and can only pick an existing action id.
Deployment (MacLustr)
deploy/social-runtime-crawler.mld.json → M1M32:~/dispatch/apps/social-runtime-crawler.json. mld stage . social-runtime-crawler && mld deploy social-runtime-crawler
places the app on the best node (Postgres 17 local social_runtime, PM2 socialcrawl-api :8350 + socialcrawl-web :8351, Chromium installed by hook) and routes
https://www.socialcrawl.co through the MacLustr Tunnel (BHS64). Browser profiles live on the node (~/apps/social-runtime-crawler/data/browser_profiles): log in once
through Screen Sharing with pnpm login <platform>. Secrets (API token, passcode) live only in the manifest on the gateway.
Layout
apps/worker CLI + crawl engine (observe → plan → act → learn loop) + action executor
apps/api control-plane API (node:http): queries, SSE stream, job runner
apps/dashboard SocialCrawl research console (Next 16 + React 19 + Tailwind 4 + recharts + d3-force)
packages/shared types (platforms, surfaces, entities, actions, budgets), utils, logger, config
packages/events Universal Social Event Format, typed bus, JSONL replay log
packages/browser SocialBrowserSession (persistent profile), interactive login
packages/observers network (NetworkObserver, ResponseClassifier, SchemaProfiler), dom (DomObserver, PageSummarizer), PageClassifier
packages/entities surface merge with provenance, confidence, IdentityResolver
packages/media video detection, frame sampler
packages/agent WorldModel, InformationGain, LoopDetector, planners, LLM clients
packages/platform-model learned platform knowledge + connector compiler
packages/storage PostgreSQL store + schema.sql
packages/connectors adapter interface, generic DOM→entity extraction, YouTube + Reddit adapters
connectors/<platform>/ compiled (learned) manifests — generated, not hand-written truth
data/ browser_profiles/ (never commit), sessions/<id>/events.jsonl, platform_model/, media/Boundaries (§4)
Only information the authenticated account can see through normal use. No bypass of authentication, access controls, CAPTCHA, rate limits, private content or privacy settings. Credentials are never typed or stored by the crawler; the browser profile is the only persisted authentication state and lives outside git.