Social Runtime Crawler
Browser-native, self-learning crawler for modern social platforms. Instead of downloading documents, it observes the runtime of a social application through an authenticated browser (Playwright + Chromium), fuses several observation surfaces (network, DOM, media, navigation), builds a world model, lets an information-gain agent decide what to look at next, and progressively learns how the platform works (response schemas, page types, action → observation patterns) into a compiled connector.
Full specification: CLAUDE.md. Architecture notes: docs/ARCHITECTURE.md.
Status — v0.1 proof of concept (2026-09-11)
Implemented (Phases 0 → 8 of §75, plus media detection and platform learning):
| Area | What works |
|---|---|
| Browser runtime | persistent Chromium profile per (platform, account), human-in-the-loop login, CDP session |
| Network surface | XHR/fetch/GraphQL/WebSocket/media capture, SchemaProfiler (field semantics without property-name assumptions), generic JSON entity miner, endpoint fingerprints |
| DOM surface | injected MutationObserver deltas by region, in-page semantic snapshot (links, articles, videos, search, expand controls) |
| Page classification | URL hints + landmarks + density, confidence, UNKNOWN never crashes |
| Entities | URL-grammar extraction, network ↔ DOM merge with per-field provenance, agreements/conflicts report, identity-resolution skeleton |
| Agent | semantic action vocabulary, information-gain scoring, lexical novelty, loop detector, heuristic planner + optional LLM planner (Anthropic or local OpenAI-compatible) |
| Media | video detection (DOM <video> + manifests/segments + video entities), frame sampling at LEVEL ≥ 2 |
| Learning | data/platform_model/<platform>/ (response, action, page, media patterns), degradation signal, connector manifest compilation |
| Storage | PostgreSQL (raw observations, entities, media, actions, schemas, feed items, relationships) + JSONL replay log |
| Dashboard | pnpm dashboard → http://localhost:8340 (decisions, page state, entities, schemas, media, connector confidence, world model, raw events) |
| Adapters | YouTube, Reddit (Phase 1). Facebook/Instagram/TikTok/X/LinkedIn/Threads: not yet |
Read-only by construction: no like/follow/comment/share/subscribe code path exists.
Quick start
pnpm install
cp .env.example .env # set SRC_DATABASE_URL (createdb social_runtime) — optional
pnpm db:migrate # creates the tables
pnpm login youtube # opens Chromium, log in by hand, press Enter → profile saved
pnpm crawl youtube --query "intelligence artificielle Québec" --goal "Discover public Quebec creators discussing AI" --minutes 10
pnpm dashboard # research console
pnpm learn youtube --minutes 15 # PLATFORM LEARNING mode + connector compilation → connectors/youtube/manifest.json
pnpm src status youtube # "Platform learned: page types / entity types / schemas / confidence"
pnpm replay # list sessions ; pnpm replay <session_id> [--type ACTION_PLANNED]
pnpm test && pnpm typecheckFlags for crawl: --mode research|observe|topic|profile|learn, --seed <url>, --minutes, --actions, --profiles, --posts, --videos, --media 0-4 (capture level, default 1), --headless, --account <alias>.
LLM planner: set SRC_LLM_PROVIDER=anthropic (uses ANTHROPIC_API_KEY or an ant auth login profile, model claude-opus-5) or SRC_LLM_PROVIDER=local with SRC_LOCAL_LLM_URL / SRC_LOCAL_LLM_MODEL (e.g. llm-api.io on the cluster). The LLM is only consulted when the heuristic is unsure, only sees the compact page state, and can only pick an existing action id.
Layout
apps/worker CLI + crawl engine (observe → plan → act → learn loop) + action executor
apps/api dashboard server (node:http) + public/index.html
packages/shared types (platforms, surfaces, entities, actions, budgets), utils, logger, config
packages/events Universal Social Event Format, typed bus, JSONL replay log
packages/browser SocialBrowserSession (persistent profile), interactive login
packages/observers network (NetworkObserver, ResponseClassifier, SchemaProfiler), dom (DomObserver, PageSummarizer), PageClassifier
packages/entities surface merge with provenance, confidence, IdentityResolver
packages/media video detection, frame sampler
packages/agent WorldModel, InformationGain, LoopDetector, planners, LLM clients
packages/platform-model learned platform knowledge + connector compiler
packages/storage PostgreSQL store + schema.sql
packages/connectors adapter interface, generic DOM→entity extraction, YouTube + Reddit adapters
connectors/<platform>/ compiled (learned) manifests — generated, not hand-written truth
data/ browser_profiles/ (never commit), sessions/<id>/events.jsonl, platform_model/, media/Boundaries (§4)
Only information the authenticated account can see through normal use. No bypass of authentication, access controls, CAPTCHA, rate limits, private content or privacy settings. Credentials are never typed or stored by the crawler; the browser profile is the only persisted authentication state and lives outside git.