spb/social-runtime-crawler

1# Architecture notes — v0.123## The loop (apps/worker/src/engine.ts)45```6seed (search url / home)            NetworkObserver.beginStep / DomObserver.beginStep7   ↓8observe()  = snapshotDom → classifyPage → entitiesFromDom ⊕ mineEntities(network) → mergeSurfaces → detectVideos → buildActions → PageState9   ↓10plan()     = scoreActions (information gain) → HeuristicPlanner | LlmPlanner (only when unsure)11   ↓12execute()  = ActionExecutor (click link / scroll / search / expand / play / back …)   ← the only place actions touch the browser13   ↓14collect network + DOM deltas of the step → observe() again → WorldModel.observe (fresh entities)15   ↓16PlatformModel.learnStep (action → responses → entities) → CONNECTOR_PATTERN_LEARNED / CONNECTOR_DEGRADED17   ↓18LoopDetector, budget, AUTH_REQUIRED checks → next step19```2021Everything is published on the `EventBus` (Universal Social Event Format) and consumed by the JSONL log (replay) and the PostgreSQL store.2223## Surfaces and provenance2425Every `ObservedEntity` carries `provenance[]` and `fields{name → {value, provenance[]}}`. The merge (`packages/entities`) keeps both network and DOM evidence, counts field agreements and conflicts, and boosts confidence when two surfaces agree. `PAGE_OPENED` events record `dom_entities / network_entities / both_surfaces / field_agreements` so the "DOM vs network" experiment (§69) is measurable from the logs.2627## Generic vs platform-specific2829Generic: browser lifecycle, network capture, `SchemaProfiler`, JSON entity miner, DOM snapshot, page classifier, planner, information gain, world model, learning, storage, dashboard.3031Adapter (`packages/connectors/<platform>.ts`): hosts, home/search URL, URL grammar → entity type + id, page-type URL hints, id key names, rich-text collapsers (`runs`/`simpleText` on YouTube), noise-link filter, expected entity counts per page type, overlay dismissal.3233Adapters contain **no endpoints and no CSS classes**. The network layer fingerprints endpoints by `(hostname, path pattern, method, response shape hash)`.3435## Learned state3637`data/platform_model/<platform>/platform_model.json` (plus split files): response patterns (shape hash → likely entity types, confidence, triggered_by action), action patterns (avg new entities, target page types, usual shapes), page types (url patterns, avg entities), media patterns. `compileConnector()` writes `connectors/<platform>/manifest.json` when confidence ≥ 40 %.3839`PlatformModel.learnedYield()` feeds the expected-entity-yield term of the information-gain formula, so navigation improves with what the crawler has learned.4041## Degradation / self-healing (first step)4243`PlatformModel.isDegraded(pageType, observed, adapterExpected)` fires `CONNECTOR_DEGRADED` when a well-known page type yields zero entities; `CONNECTOR_REPAIRED` when extraction recovers. Full re-learning (re-inspect DOM/network, regenerate the connector, run regression tests) is Phase 12.4445## Next steps46471. Run authenticated sessions on YouTube and Reddit, inspect field conflicts in the dashboard, tune adapters and the miner.482. Embeddings for novelty (local model) behind `WorldModel.novelty`.493. Facebook adapter (Phase 2) through PLATFORM LEARNING mode first: `pnpm learn facebook` once a GenericAdapter fallback exists.504. Record/replay fixtures from `sessions/<id>/events.jsonl` for regression tests of learned schemas.515. Distributed workers on MacLustr nodes (one browser profile per node), central PostgreSQL.52