# Architecture notes — v0.1

## The loop (apps/worker/src/engine.ts)

```
seed (search url / home)            NetworkObserver.beginStep / DomObserver.beginStep
   ↓
observe()  = snapshotDom → classifyPage → entitiesFromDom ⊕ mineEntities(network) → mergeSurfaces → detectVideos → buildActions → PageState
   ↓
plan()     = scoreActions (information gain) → HeuristicPlanner | LlmPlanner (only when unsure)
   ↓
execute()  = ActionExecutor (click link / scroll / search / expand / play / back …)   ← the only place actions touch the browser
   ↓
collect network + DOM deltas of the step → observe() again → WorldModel.observe (fresh entities)
   ↓
PlatformModel.learnStep (action → responses → entities) → CONNECTOR_PATTERN_LEARNED / CONNECTOR_DEGRADED
   ↓
LoopDetector, budget, AUTH_REQUIRED checks → next step
```

Everything is published on the `EventBus` (Universal Social Event Format) and consumed by the JSONL log (replay) and the PostgreSQL store.

## Surfaces and provenance

Every `ObservedEntity` carries `provenance[]` and `fields{name → {value, provenance[]}}`. The merge (`packages/entities`) keeps both network and DOM evidence, counts field agreements and conflicts, and boosts confidence when two surfaces agree. `PAGE_OPENED` events record `dom_entities / network_entities / both_surfaces / field_agreements` so the "DOM vs network" experiment (§69) is measurable from the logs.

## Generic vs platform-specific

Generic: browser lifecycle, network capture, `SchemaProfiler`, JSON entity miner, DOM snapshot, page classifier, planner, information gain, world model, learning, storage, dashboard.

Adapter (`packages/connectors/<platform>.ts`): hosts, home/search URL, URL grammar → entity type + id, page-type URL hints, id key names, rich-text collapsers (`runs`/`simpleText` on YouTube), noise-link filter, expected entity counts per page type, overlay dismissal.

Adapters contain **no endpoints and no CSS classes**. The network layer fingerprints endpoints by `(hostname, path pattern, method, response shape hash)`.

## Learned state

`data/platform_model/<platform>/platform_model.json` (plus split files): response patterns (shape hash → likely entity types, confidence, triggered_by action), action patterns (avg new entities, target page types, usual shapes), page types (url patterns, avg entities), media patterns. `compileConnector()` writes `connectors/<platform>/manifest.json` when confidence ≥ 40 %.

`PlatformModel.learnedYield()` feeds the expected-entity-yield term of the information-gain formula, so navigation improves with what the crawler has learned.

## Degradation / self-healing (first step)

`PlatformModel.isDegraded(pageType, observed, adapterExpected)` fires `CONNECTOR_DEGRADED` when a well-known page type yields zero entities; `CONNECTOR_REPAIRED` when extraction recovers. Full re-learning (re-inspect DOM/network, regenerate the connector, run regression tests) is Phase 12.

## Next steps

1. Run authenticated sessions on YouTube and Reddit, inspect field conflicts in the dashboard, tune adapters and the miner.
2. Embeddings for novelty (local model) behind `WorldModel.novelty`.
3. Facebook adapter (Phase 2) through PLATFORM LEARNING mode first: `pnpm learn facebook` once a GenericAdapter fallback exists.
4. Record/replay fixtures from `sessions/<id>/events.jsonl` for regression tests of learned schemas.
5. Distributed workers on MacLustr nodes (one browser profile per node), central PostgreSQL.