# Social Runtime Crawler

Browser-native, self-learning crawler for modern social platforms. Instead of downloading documents, it **observes the runtime of a social application** through an authenticated browser (Playwright + Chromium), fuses several observation surfaces (network, DOM, media, navigation), builds a world model, lets an information-gain agent decide what to look at next, and progressively **learns how the platform works** (response schemas, page types, action → observation patterns) into a compiled connector.

Full specification: [`CLAUDE.md`](./CLAUDE.md). Architecture notes: [`docs/ARCHITECTURE.md`](./docs/ARCHITECTURE.md).

## Status — v0.1 proof of concept (2026-09-11)

Implemented (Phases 0 → 8 of §75, plus media detection and platform learning):

| Area | What works |
|---|---|
| Browser runtime | persistent Chromium profile per (platform, account), human-in-the-loop login, CDP session |
| Network surface | XHR/fetch/GraphQL/WebSocket/media capture, `SchemaProfiler` (field semantics without property-name assumptions), generic JSON entity miner, endpoint fingerprints |
| DOM surface | injected `MutationObserver` deltas by region, in-page semantic snapshot (links, articles, videos, search, expand controls) |
| Page classification | URL hints + landmarks + density, confidence, `UNKNOWN` never crashes |
| Entities | URL-grammar extraction, network ↔ DOM merge with per-field provenance, agreements/conflicts report, identity-resolution skeleton |
| Agent | semantic action vocabulary, information-gain scoring, lexical novelty, loop detector, heuristic planner + optional LLM planner (Anthropic or local OpenAI-compatible) |
| Media | video detection (DOM `<video>` + manifests/segments + video entities), frame sampling at LEVEL ≥ 2 |
| Learning | `data/platform_model/<platform>/` (response, action, page, media patterns), degradation signal, connector manifest compilation |
| Storage | PostgreSQL (raw observations, entities, media, actions, schemas, feed items, relationships) + JSONL replay log |
| Console | **www.socialcrawl.co** — Next 16 research console (`apps/dashboard`): observatory, session console (decisions, page state, entities, runtime APIs, media, world-model graph, live events), entity evidence, runtime API explorer, connectors, mission launcher. Passcode gate. |
| Control plane | `apps/api` (node:http, port 8350): overview/sessions/entities/schemas/platforms queries, SSE `/api/stream`, job runner (spawns worker processes, `crawl_jobs`). Mutations need `SRC_API_TOKEN`. |
| Adapters | YouTube, Reddit (Phase 1), Facebook (Phase 2 — §52 prototype validated 2026-09-12 with an authenticated profile: search → group → public page → posts → profiles, GraphQL operations learned automatically). Instagram/TikTok/X/LinkedIn/Threads: not yet |

Read-only by construction: no like/follow/comment/share/subscribe code path exists.

Runtime data (`data/`: browser profiles, sessions, media) is excluded from git **and** from `mld stage` through `.mldignore` — an authenticated
profile must never leave the machine it was created on. On the laptop, `SRC_DATA_DIR` can point outside the repository.

## Quick start

```bash
pnpm install
cp .env.example .env            # set SRC_DATABASE_URL (createdb social_runtime) — optional
pnpm db:migrate                 # creates the tables

pnpm login youtube              # opens Chromium, log in by hand, press Enter → profile saved
pnpm crawl youtube --query "intelligence artificielle Québec" --goal "Discover public Quebec creators discussing AI" --minutes 10
SRC_API_PORT=8350 pnpm api      # control-plane API
pnpm dashboard:dev              # console on http://localhost:8351 (API_URL=http://127.0.0.1:8350; set SRC_DASHBOARD_PASSCODE to enable the gate)

pnpm learn youtube --minutes 15 # PLATFORM LEARNING mode + connector compilation → connectors/youtube/manifest.json
pnpm src status youtube         # "Platform learned: page types / entity types / schemas / confidence"
pnpm replay                     # list sessions ; pnpm replay <session_id> [--type ACTION_PLANNED]
pnpm test && pnpm typecheck
```

Flags for `crawl`: `--mode research|observe|topic|profile|learn`, `--seed <url>`, `--minutes`, `--actions`, `--profiles`, `--posts`, `--videos`, `--media 0-4` (capture level, default 1), `--headless`, `--account <alias>`.

LLM planner: set `SRC_LLM_PROVIDER=anthropic` (uses `ANTHROPIC_API_KEY` or an `ant auth login` profile, model `claude-opus-5`) or `SRC_LLM_PROVIDER=local` with `SRC_LOCAL_LLM_URL` / `SRC_LOCAL_LLM_MODEL` (e.g. llm-api.io on the cluster). The LLM is only consulted when the heuristic is unsure, only sees the compact page state, and can only pick an existing action id.

## Deployment (MacLustr)

`deploy/social-runtime-crawler.mld.json` → `M1M32:~/dispatch/apps/social-runtime-crawler.json`. `mld stage . social-runtime-crawler && mld deploy social-runtime-crawler`
places the app on the best node (Postgres 17 local `social_runtime`, PM2 `socialcrawl-api` :8350 + `socialcrawl-web` :8351, Chromium installed by hook) and routes
`https://www.socialcrawl.co` through the MacLustr Tunnel (BHS64). Browser profiles live on the node (`~/apps/social-runtime-crawler/data/browser_profiles`): log in once
through Screen Sharing with `pnpm login <platform>`. Secrets (API token, passcode) live only in the manifest on the gateway.

## Layout

```
apps/worker      CLI + crawl engine (observe → plan → act → learn loop) + action executor
apps/api         control-plane API (node:http): queries, SSE stream, job runner
apps/dashboard   SocialCrawl research console (Next 16 + React 19 + Tailwind 4 + recharts + d3-force)
packages/shared  types (platforms, surfaces, entities, actions, budgets), utils, logger, config
packages/events  Universal Social Event Format, typed bus, JSONL replay log
packages/browser SocialBrowserSession (persistent profile), interactive login
packages/observers  network (NetworkObserver, ResponseClassifier, SchemaProfiler), dom (DomObserver, PageSummarizer), PageClassifier
packages/entities   surface merge with provenance, confidence, IdentityResolver
packages/media      video detection, frame sampler
packages/agent      WorldModel, InformationGain, LoopDetector, planners, LLM clients
packages/platform-model  learned platform knowledge + connector compiler
packages/storage    PostgreSQL store + schema.sql
packages/connectors adapter interface, generic DOM→entity extraction, YouTube + Reddit adapters
connectors/<platform>/  compiled (learned) manifests — generated, not hand-written truth
data/            browser_profiles/ (never commit), sessions/<id>/events.jsonl, platform_model/, media/
```

## Boundaries (§4)

Only information the authenticated account can see through normal use. No bypass of authentication, access controls, CAPTCHA, rate limits, private content or privacy settings. Credentials are never typed or stored by the crawler; the browser profile is the only persisted authentication state and lives outside git.