# Architecture notes — v0.1

# The loop (apps/worker/src/engine.ts)

seed (search url / home)            NetworkObserver.beginStep / DomObserver.beginStep
   ↓
observe()  = snapshotDom → classifyPage → entitiesFromDom ⊕ mineEntities(network) → mergeSurfaces → detectVideos → buildActions → PageState
   ↓
plan()     = scoreActions (information gain) → HeuristicPlanner | LlmPlanner (only when unsure)
   ↓
execute()  = ActionExecutor (click link / scroll / search / expand / play / back …)   ← the only place actions touch the browser
   ↓
collect network + DOM deltas of the step → observe() again → WorldModel.observe (fresh entities)
   ↓
PlatformModel.learnStep (action → responses → entities) → CONNECTOR_PATTERN_LEARNED / CONNECTOR_DEGRADED
   ↓
LoopDetector, budget, AUTH_REQUIRED checks → next step

Everything is published on the EventBus (Universal Social Event Format) and consumed by the JSONL log (replay) and the PostgreSQL store.

# Surfaces and provenance

Every ObservedEntity carries provenance[] and fields{name → {value, provenance[]}}. The merge (packages/entities) keeps both network and DOM evidence, counts field agreements and conflicts, and boosts confidence when two surfaces agree. PAGE_OPENED events record dom_entities / network_entities / both_surfaces / field_agreements so the "DOM vs network" experiment (§69) is measurable from the logs.

# Generic vs platform-specific

Generic: browser lifecycle, network capture, SchemaProfiler, JSON entity miner, DOM snapshot, page classifier, planner, information gain, world model, learning, storage, dashboard.

Adapter (packages/connectors/<platform>.ts): hosts, home/search URL, URL grammar → entity type + id, page-type URL hints, id key names, rich-text collapsers (runs/simpleText on YouTube), noise-link filter, expected entity counts per page type, overlay dismissal.

Adapters contain no endpoints and no CSS classes. The network layer fingerprints endpoints by (hostname, path pattern, method, response shape hash).

# Learned state

data/platform_model/<platform>/platform_model.json (plus split files): response patterns (shape hash → likely entity types, confidence, triggered_by action), action patterns (avg new entities, target page types, usual shapes), page types (url patterns, avg entities), media patterns. compileConnector() writes connectors/<platform>/manifest.json when confidence ≥ 40 %.

PlatformModel.learnedYield() feeds the expected-entity-yield term of the information-gain formula, so navigation improves with what the crawler has learned.

# Degradation / self-healing (first step)

PlatformModel.isDegraded(pageType, observed, adapterExpected) fires CONNECTOR_DEGRADED when a well-known page type yields zero entities; CONNECTOR_REPAIRED when extraction recovers. Full re-learning (re-inspect DOM/network, regenerate the connector, run regression tests) is Phase 12.

# Next steps

1. Run authenticated sessions on YouTube and Reddit, inspect field conflicts in the dashboard, tune adapters and the miner.
2. Embeddings for novelty (local model) behind WorldModel.novelty.
3. Facebook adapter (Phase 2) through PLATFORM LEARNING mode first: pnpm learn facebook once a GenericAdapter fallback exists.
4. Record/replay fixtures from sessions/<id>/events.jsonl for regression tests of learned schemas.
5. Distributed workers on MacLustr nodes (one browser profile per node), central PostgreSQL.