SPB Git
spboucher.ai

spb/social-runtime-crawler

Public

Social Runtime Crawler v0.1 — browser-native self-learning social crawler PoC

Monorepo (pnpm, strict TS via tsx, vitest): persistent Chromium sessions with human login,
network observer + SchemaProfiler + generic JSON entity miner, DOM mutation observer + in-page
semantic snapshot, page classifier, network/DOM surface merge with provenance, video detection
and frame sampling, world model, information-gain heuristic planner (+ optional Anthropic/local
LLM planner), loop detection, platform-model learning and connector compilation, PostgreSQL +
JSONL replay storage, research dashboard, YouTube and Reddit adapters.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Sep 12, 2026) Browse files

67 changed files +6,657 −0

67 changed files
Load all 67 files Unified Split
added .env.example +27 −0
@@ -0,0 +1,27 @@
1+# Social Runtime Crawler — environment
2+# Copy to .env (never commit .env). No platform passwords belong here or anywhere in the repo:
3+# authentication is done by a human once in the headed browser (`pnpm login <platform>`).
4+
5+# PostgreSQL (prototype storage). Leave empty to run with the JSONL-only store.
6+SRC_DATABASE_URL=postgres://localhost:5432/social_runtime
7+
8+# Where browser profiles, session logs and learned platform models live.
9+SRC_DATA_DIR=./data
10+
11+# LLM planner (Tier 4). If no provider is configured the heuristic planner (Tier 1) is used alone.
12+# anthropic → uses ANTHROPIC_API_KEY (or an `ant auth login` profile)
13+# local → OpenAI-compatible local endpoint (e.g. llm-api.io on the MacLustr cluster)
14+# none → heuristic only
15+SRC_LLM_PROVIDER=none
16+SRC_LLM_MODEL=claude-opus-5
17+# ANTHROPIC_API_KEY=
18+# SRC_LOCAL_LLM_URL=https://www.llm-api.io/v1
19+# SRC_LOCAL_LLM_KEY=
20+# SRC_LOCAL_LLM_MODEL=
21+
22+# Browser
23+SRC_HEADLESS=false
24+SRC_BROWSER_CHANNEL=chromium
25+
26+# Dashboard
27+SRC_DASHBOARD_PORT=8340
added .gitignore +16 −0
@@ -0,0 +1,16 @@
1+node_modules/
2+dist/
3+.env
4+.env.local
5+*.log
6+.DS_Store
7+
8+# Runtime data: browser profiles hold authenticated sessions — never commit.
9+data/browser_profiles/
10+data/sessions/
11+data/media/
12+data/platform_model/*/
13+!data/platform_model/.gitkeep
14+
15+# Compiled (learned) connectors are versioned intentionally; regression fixtures may be large.
16+connectors/*/regression_tests/large/
added CLAUDE.md +332 −0
@@ -0,0 +1,332 @@
1+# CLAUDE.md
2+
3+## Project Name
4+
5+**Social Runtime Crawler**
6+
7+Alternative internal names: SRC · SocialProbe · Social Runtime Mining · Social Browser Intelligence Engine · Living Social Crawler
8+
9+> Repository guide for contributors (humans and agents). Sections 1–87 are the founding specification.
10+> Implementation status, commands and layout: see `README.md` and `docs/ARCHITECTURE.md`.
11+> Repo conventions: pnpm workspaces, strict TypeScript run with `tsx` (no build step), vitest, `@src/*` packages, `pnpm typecheck && pnpm test` before committing.
12+> In-page code (evaluated in the browser) must be plain JavaScript strings or `.page.js` files — TS transpilers inject helpers (`__name`) that do not exist in the page.
13+> Never put credentials in the repo or in `.env`; authentication = human login into the persistent browser profile (`pnpm login <platform>`), profiles live in `data/browser_profiles/` (git-ignored).
14+
15+---
16+
17+# 1. Mission
18+
19+Build a new browser-native crawling system specifically designed for modern social-media platforms.
20+
21+This project must NOT behave like a traditional scraper that primarily performs direct HTTP fetches against static pages.
22+
23+Instead, it must treat each social platform as a **live interactive application runtime**.
24+
25+The crawler must:
26+
27+1. launch and maintain authenticated browser sessions;
28+2. navigate social-media interfaces normally;
29+3. observe what the logged-in account is legitimately allowed to see;
30+4. intercept structured information delivered to the browser frontend;
31+5. inspect DOM mutations;
32+6. inspect network responses;
33+7. inspect browser runtime state;
34+8. detect dynamically loaded posts, profiles, videos, comments, pages and recommendations;
35+9. capture media metadata and optionally public media artifacts;
36+10. convert observations into normalized structured events;
37+11. build an internal world model of each social platform;
38+12. use an AI agent to decide what to explore next;
39+13. learn how each platform works during exploration;
40+14. progressively compile learned behavior into reusable platform connectors;
41+15. detect connector breakage and return to exploration mode automatically.
42+
43+The long-term objective is to invent a new crawling paradigm: **Social Runtime Mining**. Instead of downloading documents from the Web, the system observes the execution of social applications and learns how information is surfaced through normal browser interaction.
44+
45+# 2. Core Principle
46+
47+Traditional crawler: `URL → HTTP GET → HTML → Parse → Follow links → Repeat`.
48+
49+Social Runtime Crawler:
50+
51+```text
52+Authenticated Browser → Live Social Application → (DOM | Network | Visual)
53+→ Runtime Observation Layer → Semantic World Model → AI Navigation Agent → Action Selection
54+→ click / scroll / search / open profile / open post / play video / expand comments ↻
55+```
56+
57+The crawler should continuously ask: *What action will reveal the most useful new information?* — not *What URL should I download next?*
58+
59+# 3. Scope
60+
61+Initial target platforms: Facebook, Instagram, TikTok, X / Twitter, YouTube, Reddit, LinkedIn, Threads.
62+
63+Prototype order — Phase 1: YouTube, Reddit · Phase 2: Facebook, Instagram · Phase 3: TikTok, X · Phase 4: LinkedIn, Threads. The architecture must nevertheless be platform-independent.
64+
65+# 4. Important Boundary
66+
67+The system must only process information visible to the authenticated account through normal use of the platform.
68+
69+The system must NOT be designed to: bypass authentication; defeat access controls; obtain private messages; access private profiles not visible to the account; discover hidden API credentials; defeat CAPTCHA systems; evade platform security mechanisms; bypass rate limits through deceptive identity rotation; exploit undocumented vulnerabilities; circumvent paywalls or privacy settings; impersonate users without authorization.
70+
71+Authentication state should be supplied manually during development. The crawler may observe and structure data already delivered to the browser.
72+
73+# 5. Primary Use Case
74+
75+A research account is logged into Facebook. The crawler: open Facebook → observe feed → detect visible post → extract post metadata → detect author/page/profile → detect media → detect interactions → detect comments if opened → follow visible public profile → inspect profile → discover other public posts → return to feed → continue.
76+
77+The same architecture must work for TikTok For You, Instagram Explore, YouTube recommendations, Reddit feeds, Facebook pages and public profiles, X timelines, LinkedIn public posts.
78+
79+# 6. Architectural Philosophy
80+
81+Build this as a set of cooperating systems. Never create a monolithic Playwright script.
82+
83+Main systems: Browser Runtime · Observation Layer · Entity Extraction Layer · Media Intelligence Layer · World Model · Navigation Agent · Information Gain Engine · Platform Adapter · Connector Learning Engine · Connector Compiler · Persistence Layer · Scheduler · Replay System · Monitoring · Research UI.
84+
85+# 7. Browser Runtime
86+
87+Node.js / TypeScript, Playwright, Chromium, Chrome DevTools Protocol. Primary controller: Playwright. Lower-level telemetry: CDP.
88+
89+Each account uses a persistent browser profile (`browser_profiles/<platform>-<alias>/`). Never persist passwords manually in project files. Authentication: launch browser → human login → save browser profile → reuse authenticated profile.
90+
91+# 8. Browser Sessions
92+
93+```ts
94+interface SocialBrowserSession {
95+ sessionId: string; platform: Platform; profilePath: string; accountAlias: string;
96+ start(): Promise<void>; stop(): Promise<void>; navigate(url: string): Promise<void>; getCurrentState(): Promise<PageState>;
97+}
98+```
99+
100+Track: session_id, platform, account_alias, started_at, current_url, current_entity, navigation_depth, last_action, health.
101+
102+# 9. Observation Surfaces
103+
104+SURFACE_1_NETWORK · SURFACE_2_DOM · SURFACE_3_RUNTIME_STATE · SURFACE_4_ACCESSIBILITY · SURFACE_5_VISUAL · SURFACE_6_MEDIA · SURFACE_7_NAVIGATION.
105+
106+Each observation must contain provenance: `{ "value": "Example Creator", "source": "network", "confidence": 0.99, "observed_at": "..." }`.
107+
108+# 10. Network Observation Layer
109+
110+Observe browser-generated XHR, fetch, GraphQL, JSON, HTML fragments, WebSocket frames, SSE, media manifests, video metadata, image resources. Modules: NetworkObserver, ResponseClassifier, JsonDetector, GraphQLDetector, WebSocketObserver, MediaRequestObserver, SchemaProfiler.
111+
112+Do NOT hardcode endpoints during initial exploration. Fingerprint them: `{ hostname, method, content_type, response_shape_hash, observed_entity_types }`.
113+
114+# 11. Automatic Response Classification
115+
116+response → is JSON? → schema inference → contains repeated objects? → candidate entity detection → field analysis → relationship detection → confidence score.
117+
118+Detect post IDs, profile IDs, usernames, captions, timestamps, engagement counts, comment arrays, cursor values, pagination tokens, media URLs, thumbnail URLs, video manifests, author objects, recommendation metadata. Avoid assumptions about property names; use structure and semantic inference.
119+
120+# 12. Schema Discovery
121+
122+`SchemaProfiler` infers `abc → likely_identifier`, `xyz.foo → likely_display_name`, `xyz.bar → likely_username` from field name, value shape, frequency, co-occurrence, DOM evidence, visual evidence, LLM semantic inference.
123+
124+# 13. DOM Observation
125+
126+MutationObserver inside the page. Track elements added/removed, text changes, attributes, lazy-loaded sections, new feed cards, modal opening, comment expansion, video replacement, infinite-scroll additions. Inject a lightweight observer; emit deltas (`{ "event": "dom_nodes_added", "count": 14, "region": "feed" }`), never serialize the entire DOM continuously.
127+
128+# 14. Semantic DOM Representation
129+
130+Never send the full DOM blindly to an LLM. Generate a compact page representation (PAGE / VISIBLE ENTITIES [1] Page… [2] Post… / ACTIONS [A1]…). The AI agent chooses only semantic actions.
131+
132+# 15. Accessibility Surface
133+
134+Optional observer using accessible roles and labels (buttons, links, menus, tabs, search fields, video controls, follow buttons, profile links, comment controls). Prefer semantic locators over fragile generated CSS classes.
135+
136+# 16. Visual Surface
137+
138+Fallback, not primary. Capture viewport screenshots, content cards, video frames, profile headers, overlays — only when network confidence low, DOM confidence low, content image-based, text embedded in media, UI unknown, or video classification required.
139+
140+# 17. Video Intelligence
141+
142+`src/media/video/`: VideoDetector, VideoMetadataExtractor, VideoFrameSampler, SubtitleExtractor, TranscriptResolver, AudioMetadataExtractor, VideoFingerprint, MediaDeduplicator. Collect platform video ID, author, caption, description, hashtags, duration, thumbnail, public engagement metrics, creation timestamp, visible music/audio name, subtitles, resolution, delivery metadata.
143+
144+# 18. Video Frame Sampling
145+
146+Do NOT download every video. video detected → metadata → relevance estimate → if relevant: capture frames (start, 25 %, 50 %, 75 %, end; adaptive: scene change) → vision analysis → semantic representation. Store perceptual hashes for dedup.
147+
148+# 19. Video Semantic Record
149+
150+Normalized `{ media_type, platform, platform_media_id, author_entity_id, caption, topics[], entities[], language, transcript_available, visual_summary, engagement{views,likes,comments} }`.
151+
152+# 20. Public Profile Modeling
153+
154+Profiles become canonical entities `{ entity_type: "person", canonical_id, display_name, platform_profiles{}, professional_context, public_bio, topics[], sources[] }`. Do not infer sensitive attributes (religion, medical conditions, sexual orientation, private family information, home address, political ideology inferred from behavior) unless scope explicitly changes with a justified basis.
155+
156+# 21. Identity Resolution
157+
158+`IdentityResolver` signals: exact name, username similarity, verified links, official website, cross-linked accounts, same organization, bio similarity, profile photo similarity where appropriate, public external links. Never merge two people solely because names match; every merge requires confidence and evidence.
159+
160+# 22. Entity Types
161+
162+Person, Organization, Page, Profile, Channel, Account, Post, Comment, Video, Image, Topic, Hashtag, URL, Event, Location, Product, OrganizationRole. Relationships: AUTHORED, MENTIONED, REPLIED_TO, POSTED_BY, BELONGS_TO, LINKS_TO, FEATURES, HAS_PROFILE, REPRESENTS, DISCOVERED_FROM.
163+
164+# 23. Social World Model
165+
166+Graph-like in-memory representation of the current platform (Profile A —AUTHORED→ Post 1 …). This graph guides navigation.
167+
168+# 24. Navigation Agent
169+
170+The LLM must NOT manipulate raw selectors. It receives `{ goal, current_state, visible_entities[], available_actions[] }` and answers `{ action, expected_information_gain, reason }`.
171+
172+# 25. Action Vocabulary
173+
174+OPEN_ENTITY, OPEN_POST, OPEN_PROFILE, OPEN_PAGE, OPEN_CHANNEL, OPEN_VIDEO, OPEN_COMMENTS, SCROLL_DOWN, SCROLL_UP, SEARCH, FILTER, PLAY_VIDEO, PAUSE_VIDEO, EXPAND, COLLAPSE, BACK, FORWARD, RETURN_TO_FEED, WAIT_FOR_CONTENT, END_SESSION. Do not expose arbitrary browser JavaScript to the planner.
175+
176+# 26. Information Gain Engine
177+
178+`information_gain = novelty × relevance × expected_entity_yield × confidence × source_quality ÷ exploration_cost`, with penalties: already_seen, duplicate_content, navigation_loop, low_quality_source, low_relevance.
179+
180+# 27. Novelty Measurement
181+
182+Embeddings: novelty ≈ `1 − max_similarity(candidate, existing dataset)`. Use a local embedding model whenever possible.
183+
184+# 28. Agent Modes
185+
186+OBSERVE (feed only; study recommendation exposure) · RESEARCH (search + navigate public entities) · PROFILE (one public person) · TOPIC (a topic → people, organizations, posts, videos, channels, hashtags) · PLATFORM LEARNING (understand UI, runtime, entity structures, schemas, navigation).
187+
188+# 29. Connector Learning
189+
190+Do not immediately hardcode a Facebook connector; learn first. Store `platform_model/{page_types, entity_patterns, response_patterns, action_patterns, navigation_graph, selectors, media_patterns}.json`.
191+
192+# 30. Action → Observation Learning
193+
194+Every action records consequences (`before`/`after`: visible posts, network requests, new entities). Infer over time: `scroll_down → usually FeedResponseType3 → ~8 new posts → exposes cursor`.
195+
196+# 31. Automatic Connector Compilation
197+
198+Once confidence is high: learn platform → compile connector (`connectors/<platform>/manifest, page_classifier, network_patterns, entities, navigation, media, parser, regression_tests/`). The connector is learned state, not manually written truth.
199+
200+# 32. Self-Healing
201+
202+`expected entities = 20, observed = 0` → `CONNECTOR_DEGRADED` → exploration mode → re-inspect DOM/network → re-learn → update connector → regression tests → resume.
203+
204+# 33. Universal Social Event Format
205+
206+`{ event_id, event_type, platform, session_id, timestamp, entity{type, platform_id, canonical_id}, provenance[{surface, confidence}], discovered_via{action} }`.
207+
208+# 34. Event Types
209+
210+PAGE_OPENED · ENTITY_DISCOVERED/OBSERVED/UPDATED · POST_/COMMENT_/PROFILE_DISCOVERED · MEDIA_/VIDEO_/IMAGE_DISCOVERED · NETWORK_RESPONSE_OBSERVED · NETWORK_SCHEMA_DISCOVERED · DOM_CHANGED · ACTION_EXECUTED/FAILED · NAVIGATION_COMPLETED · CONNECTOR_PATTERN_LEARNED/DEGRADED/REPAIRED.
211+
212+# 35. Storage
213+
214+Prototype: PostgreSQL, Redis, local filesystem / S3-compatible object storage. Optional later: ClickHouse, OpenSearch, Qdrant, Neo4j. Do not add all databases prematurely.
215+
216+# 36. PostgreSQL Core Tables
217+
218+sessions, platform_accounts, entities, entity_aliases, observations, posts, media, videos, relationships, actions, network_responses, schema_patterns, connector_versions, crawl_jobs.
219+
220+# 37. Raw vs Normalized Data
221+
222+RAW OBSERVATION → NORMALIZATION → CANONICAL ENTITY. Raw observations must never be overwritten.
223+
224+# 38. Evidence Model
225+
226+Every canonical field needs provenance: `{ field, value, evidence[{observation_id, source}] }`.
227+
228+# 39. Deduplication
229+
230+Same platform ID, canonical URL, media fingerprint, normalized text, high embedding similarity, same author+timestamp+content. Never deduplicate people using fuzzy names alone.
231+
232+# 40. Crawl Budget
233+
234+`budget: { max_minutes, max_actions, max_profiles, max_posts, max_videos, max_depth }` plus max_video_processing_seconds, max_storage_mb, max_llm_tokens.
235+
236+# 41. Navigation Safety
237+
238+Maintain visited_entities, visited_urls, recent_actions, navigation_path, page_fingerprints. Detect loops (A → B → A → B) and break automatically.
239+
240+# 42. Platform Boundary
241+
242+`stay_on_platform = true`: external links may be recorded but not followed automatically.
243+
244+# 43. Search-Based Discovery
245+
246+Research mode supports native platform search: find search field → submit query → classify result tabs → inspect people/pages/posts/videos → select relevant entities.
247+
248+# 44–45. Feed-Based Discovery & Recommendation Dataset
249+
250+Preserve ranking: feed_session_id, feed_position, content_id, timestamp_seen, time_visible, interaction_performed, recommendation_context (`is_followed_account`, `is_sponsored`, `time_visible_ms`).
251+
252+# 46. Interaction Policy
253+
254+Default **read-only**. Allowed: scroll, search, open, expand, play video, navigate. Never automatically like, follow, comment, share, message, react, subscribe unless explicitly enabled for a controlled experiment.
255+
256+# 47. Media Capture Strategy
257+
258+LEVEL 0 metadata · 1 + thumbnail · 2 + selected frames (default) · 3 + transcript/subtitles · 4 full artifact only when required and permitted.
259+
260+# 48–50. Content Classification, Local AI, Model Hierarchy
261+
262+Local LLM/embeddings for topic, language, mentions, names, locations, category, summary, tags — never invent facts. Pipeline: deterministic parser → local embeddings → local classifier → larger LLM only where needed. Tiers: 1 rules/parsers · 2 embeddings · 3 small local LLM · 4 large reasoning model. Escalate only when needed.
263+
264+# 51. Initial Prototype Goal
265+
266+YouTube or Reddit first: start authenticated browser → search topic → detect result entities → navigate results → intercept network → compare network vs DOM extraction → capture posts/videos → normalize events → store in PostgreSQL → dashboard.
267+
268+# 52. Facebook Experimental Prototype
269+
270+Open authenticated Facebook → one known public page → observe → identify posts → intercept runtime traffic → open one post → expand visible comments → identify public profile/page links → open one → capture public info → return. Success: no Facebook-specific external API, no manual data entry, browser stays authenticated, all entities have provenance, actions recorded, video objects identified, schemas discovered automatically.
271+
272+# 53. Debug UI
273+
274+Panels: Browser Preview, Current Page State, Visible Entities, Available Actions, Agent Decision, Network Events, Detected Schemas, World Model, Media Queue, Recent Observations, Connector Confidence.
275+
276+# 54–55. Replay & Explainability
277+
278+Every session replayable from recorded observations (actions, URL changes, network metadata, DOM observations, screenshots, decisions). Each AI action stores goal, chosen action, expected gain, relevance, novelty, concise reason — no hidden chain-of-thought.
279+
280+# 56. Platform Adapter Interface
281+
282+`detectPageType, extractVisibleEntities, getAvailableActions, resolveEntityTarget, detectMedia, detectFeedItems, normalizeObservation`.
283+
284+# 57. Generic Engine vs Platform Logic
285+
286+Generic: browser lifecycle, network capture, DOM observation, event bus, planning, information gain, storage, media pipeline, identity resolution. Platform-specific: page recognition, semantic labels, feed identification, known entity classes, navigation hints, normalization quirks.
287+
288+# 58–60. Page Classification, Confidence, Unknown States
289+
290+Types: HOME_FEED, SEARCH_RESULTS, PROFILE, PUBLIC_PAGE, POST_DETAIL, VIDEO_DETAIL, CHANNEL, GROUP, COMMENT_VIEW, UNKNOWN. Signals: URL, visible text, ARIA landmarks, DOM structure, network schemas, adapter hints. Every inference carries confidence; below threshold → stronger classifier. Unknown pages are expected: capture semantic DOM + network summary → classify → generic actions → learn.
291+
292+# 61–63. Connector Knowledge Base, Shared Learning, Workers
293+
294+Persist learned behaviour (`{pattern, platform, schema_hash, likely_entity_type, confidence, observed_count}`). Workers share patterns through a Connector Knowledge Service → versioned platform model. Worker = Browser + Playwright + CDP + Observer + Media Processor + Action Executor; control plane = Scheduler, Queue, DB, Connector Knowledge, LLM Router, Dashboard.
295+
296+# 64–66. Cluster Deployment, Isolation, Crash Recovery
297+
298+One Mac node per platform worker (MacLustr), central PostgreSQL/Redis/object storage/API/dashboard. One browser profile per account; separate processes. Recover from browser/tab crash, timeouts, missing selectors, session expiration (pause job, `AUTH_REQUIRED`, manual re-auth — never bypass), unexpected modals, redesigns.
299+
300+# 67–69. Observability, Quality Metrics, Experimentation
301+
302+Track actions/min, entities/min, posts/min, videos/min, responses/min, duplicate ratio, LLM calls, average gain, connector confidence, CPU/RAM, storage, errors. KPIs: entity/field precision, recall on visible page, navigation success, duplicate rate, connector stability, media detection accuracy, cost per 1 000 entities, entities per browser-hour. Compare DOM only / Network only / DOM+Network / +Agent / +Vision.
303+
304+# 70–74. Research Question & Innovations
305+
306+Can this outperform traditional social crawling (less connector code, automatic adaptation, dynamic content, recommendation context, media, learned frontend APIs, surviving UI changes)? Innovations: Runtime API Discovery (action → response → entity), Social Runtime Graph (UI states, actions, schemas), Information-Gain Navigation, Connector Compiler (`social-runtime learn facebook` → "Platform learned … Confidence: 94 %"; `social-runtime crawl facebook --goal "Quebec public personalities"`).
307+
308+# 75. Development Phases
309+
310+0 repo · 1 persistent browser · 2 network observer · 3 DOM observer · 4 event normalization · 5 entity extraction · 6 semantic actions · 7 AI planner · 8 information gain · 9 media/video · 10 platform learning · 11 connector compilation · 12 self-healing · 13 distributed workers.
311+
312+# 76–77. Repository Structure & Stack
313+
314+`apps/{api,dashboard,worker}`, `packages/{browser,observers,agent,entities,media,connectors,platform-model,storage,events,shared}`, `connectors/<platform>/`, `data/`, `scripts/`, `docs/`, `docker/`. TypeScript, Node.js, Playwright, Chromium, CDP, PostgreSQL, Redis, React/Next.js dashboard, Docker where appropriate; optional Qdrant, ClickHouse, Neo4j, MinIO. Do not over-engineer the prototype.
315+
316+# 78–81. Coding Principles & Testing
317+
318+Strict TypeScript, small modules, typed events, DI where useful, structured logging, deterministic parsers before LLMs, clear adapters, unit + integration tests, record/replay fixtures. Avoid one giant platform script, hundreds of static CSS selectors, massive switch statements, platform logic inside the core, LLM-controlled arbitrary JS. Fixture-driven tests for schemas, DOM fragments, normalization, identity resolution, action selection, media detection, page classification; browser integration tests for profile reuse, navigation, scroll, search, open entity, interception, mutation capture, media detection.
319+
320+# 82–86. Data Review, Vision, Success
321+
322+Dashboard lets a human inspect raw observation, normalized value, canonical entity, evidence, surface, confidence and correct mappings (feeding learning). Vision: `"Find public Quebec personalities discussing AI"` → system understands platforms, navigates, observes runtime, discovers profiles, captures posts/videos, extracts entities, deduplicates identities, builds graph, returns dataset — without a hand-maintained scraper per page.
323+
324+First milestone succeeds when: one human login is reused; navigation by semantic actions; dynamic content observed; posts/profiles/media recognized; useful network payloads identified; network vs DOM evidence compared; video metadata captured; normalized events stored; next action chosen; loops avoided; at least one reusable platform pattern learned automatically.
325+
326+# 87. Final Engineering Rule
327+
328+When implementing any feature, always ask: *Are we teaching the crawler how the social application works, or merely hardcoding another scraper?*
329+
330+Prefer learning · observation · semantic actions · structured frontend data legitimately delivered to the browser · deterministic extraction over LLM guessing · provenance over assumptions · reusable platform knowledge over one-off automation.
331+
332+The end goal is not to automate Chrome. It is to **build an autonomous system capable of understanding and mining the observable runtime of social applications.**
added README.md +69 −0
@@ -0,0 +1,69 @@
1+# Social Runtime Crawler
2+
3+Browser-native, self-learning crawler for modern social platforms. Instead of downloading documents, it **observes the runtime of a social application** through an authenticated browser (Playwright + Chromium), fuses several observation surfaces (network, DOM, media, navigation), builds a world model, lets an information-gain agent decide what to look at next, and progressively **learns how the platform works** (response schemas, page types, action → observation patterns) into a compiled connector.
4+
5+Full specification: [`CLAUDE.md`](./CLAUDE.md). Architecture notes: [`docs/ARCHITECTURE.md`](./docs/ARCHITECTURE.md).
6+
7+## Status — v0.1 proof of concept (2026-09-11)
8+
9+Implemented (Phases 0 → 8 of §75, plus media detection and platform learning):
10+
11+| Area | What works |
12+|---|---|
13+| Browser runtime | persistent Chromium profile per (platform, account), human-in-the-loop login, CDP session |
14+| Network surface | XHR/fetch/GraphQL/WebSocket/media capture, `SchemaProfiler` (field semantics without property-name assumptions), generic JSON entity miner, endpoint fingerprints |
15+| DOM surface | injected `MutationObserver` deltas by region, in-page semantic snapshot (links, articles, videos, search, expand controls) |
16+| Page classification | URL hints + landmarks + density, confidence, `UNKNOWN` never crashes |
17+| Entities | URL-grammar extraction, network ↔ DOM merge with per-field provenance, agreements/conflicts report, identity-resolution skeleton |
18+| Agent | semantic action vocabulary, information-gain scoring, lexical novelty, loop detector, heuristic planner + optional LLM planner (Anthropic or local OpenAI-compatible) |
19+| Media | video detection (DOM `<video>` + manifests/segments + video entities), frame sampling at LEVEL ≥ 2 |
20+| Learning | `data/platform_model/<platform>/` (response, action, page, media patterns), degradation signal, connector manifest compilation |
21+| Storage | PostgreSQL (raw observations, entities, media, actions, schemas, feed items, relationships) + JSONL replay log |
22+| Dashboard | `pnpm dashboard` → http://localhost:8340 (decisions, page state, entities, schemas, media, connector confidence, world model, raw events) |
23+| Adapters | YouTube, Reddit (Phase 1). Facebook/Instagram/TikTok/X/LinkedIn/Threads: not yet |
24+
25+Read-only by construction: no like/follow/comment/share/subscribe code path exists.
26+
27+## Quick start
28+
29+```bash
30+pnpm install
31+cp .env.example .env # set SRC_DATABASE_URL (createdb social_runtime) — optional
32+pnpm db:migrate # creates the tables
33+
34+pnpm login youtube # opens Chromium, log in by hand, press Enter → profile saved
35+pnpm crawl youtube --query "intelligence artificielle Québec" --goal "Discover public Quebec creators discussing AI" --minutes 10
36+pnpm dashboard # research console
37+
38+pnpm learn youtube --minutes 15 # PLATFORM LEARNING mode + connector compilation → connectors/youtube/manifest.json
39+pnpm src status youtube # "Platform learned: page types / entity types / schemas / confidence"
40+pnpm replay # list sessions ; pnpm replay <session_id> [--type ACTION_PLANNED]
41+pnpm test && pnpm typecheck
42+```
43+
44+Flags for `crawl`: `--mode research|observe|topic|profile|learn`, `--seed <url>`, `--minutes`, `--actions`, `--profiles`, `--posts`, `--videos`, `--media 0-4` (capture level, default 1), `--headless`, `--account <alias>`.
45+
46+LLM planner: set `SRC_LLM_PROVIDER=anthropic` (uses `ANTHROPIC_API_KEY` or an `ant auth login` profile, model `claude-opus-5`) or `SRC_LLM_PROVIDER=local` with `SRC_LOCAL_LLM_URL` / `SRC_LOCAL_LLM_MODEL` (e.g. llm-api.io on the cluster). The LLM is only consulted when the heuristic is unsure, only sees the compact page state, and can only pick an existing action id.
47+
48+## Layout
49+
50+```
51+apps/worker CLI + crawl engine (observe → plan → act → learn loop) + action executor
52+apps/api dashboard server (node:http) + public/index.html
53+packages/shared types (platforms, surfaces, entities, actions, budgets), utils, logger, config
54+packages/events Universal Social Event Format, typed bus, JSONL replay log
55+packages/browser SocialBrowserSession (persistent profile), interactive login
56+packages/observers network (NetworkObserver, ResponseClassifier, SchemaProfiler), dom (DomObserver, PageSummarizer), PageClassifier
57+packages/entities surface merge with provenance, confidence, IdentityResolver
58+packages/media video detection, frame sampler
59+packages/agent WorldModel, InformationGain, LoopDetector, planners, LLM clients
60+packages/platform-model learned platform knowledge + connector compiler
61+packages/storage PostgreSQL store + schema.sql
62+packages/connectors adapter interface, generic DOM→entity extraction, YouTube + Reddit adapters
63+connectors/<platform>/ compiled (learned) manifests — generated, not hand-written truth
64+data/ browser_profiles/ (never commit), sessions/<id>/events.jsonl, platform_model/, media/
65+```
66+
67+## Boundaries (§4)
68+
69+Only information the authenticated account can see through normal use. No bypass of authentication, access controls, CAPTCHA, rate limits, private content or privacy settings. Credentials are never typed or stored by the crawler; the browser profile is the only persisted authentication state and lives outside git.
added apps/api/package.json +10 −0
@@ -0,0 +1,10 @@
1+{
2+ "name": "@src/api",
3+ "version": "0.1.0",
4+ "private": true,
5+ "type": "module",
6+ "dependencies": {
7+ "@src/shared": "workspace:*",
8+ "@src/storage": "workspace:*"
9+ }
10+}
added apps/api/public/index.html +146 −0
@@ -0,0 +1,146 @@
1+<!doctype html>
2+<html lang="en">
3+<head>
4+<meta charset="utf-8" />
5+<title>Social Runtime Crawler — research console</title>
6+<meta name="viewport" content="width=device-width, initial-scale=1" />
7+<style>
8+ :root { --bg:#0b0d10; --panel:#12161b; --line:#1f262e; --fg:#d7dee6; --dim:#7f8b97; --acc:#4cc2ff; --ok:#41d18d; --warn:#ffb454; --bad:#ff6b6b; }
9+ * { box-sizing:border-box; }
10+ body { margin:0; background:var(--bg); color:var(--fg); font:13px/1.45 "SF Mono", ui-monospace, Menlo, monospace; }
11+ header { display:flex; gap:16px; align-items:center; padding:10px 16px; border-bottom:1px solid var(--line); position:sticky; top:0; background:var(--bg); z-index:2; }
12+ header h1 { font-size:14px; margin:0; letter-spacing:.04em; color:var(--acc); }
13+ header select, header button { background:var(--panel); color:var(--fg); border:1px solid var(--line); padding:5px 8px; font:inherit; border-radius:4px; }
14+ main { display:grid; grid-template-columns: repeat(12, 1fr); gap:10px; padding:10px 16px; }
15+ section { background:var(--panel); border:1px solid var(--line); border-radius:6px; min-height:120px; display:flex; flex-direction:column; overflow:hidden; }
16+ section h2 { margin:0; padding:7px 10px; font-size:11px; text-transform:uppercase; letter-spacing:.08em; color:var(--dim); border-bottom:1px solid var(--line); display:flex; justify-content:space-between; }
17+ section .body { padding:8px 10px; overflow:auto; max-height:420px; white-space:pre-wrap; word-break:break-word; }
18+ .c3 { grid-column: span 3; } .c4 { grid-column: span 4; } .c6 { grid-column: span 6; } .c8 { grid-column: span 8; } .c12 { grid-column: span 12; }
19+ table { width:100%; border-collapse:collapse; } td, th { padding:3px 6px; border-bottom:1px solid var(--line); text-align:left; vertical-align:top; } th { color:var(--dim); font-weight:normal; }
20+ .tag { display:inline-block; padding:0 6px; border-radius:3px; background:#1a222b; color:var(--acc); margin-right:4px; font-size:11px; }
21+ .ok { color:var(--ok);} .warn { color:var(--warn);} .bad { color:var(--bad);} .dim { color:var(--dim);}
22+ .kpi { display:flex; gap:18px; flex-wrap:wrap; } .kpi div b { display:block; font-size:20px; color:var(--fg); } .kpi div { color:var(--dim); }
23+ .bar { height:6px; background:#1a222b; border-radius:3px; overflow:hidden; } .bar i { display:block; height:100%; background:var(--acc); }
24+ a { color:var(--acc); text-decoration:none; }
25+ .frames img { height:70px; margin:2px; border:1px solid var(--line); border-radius:3px; }
26+</style>
27+</head>
28+<body>
29+<header>
30+ <h1>SOCIAL RUNTIME CRAWLER</h1>
31+ <select id="session"></select>
32+ <button id="refresh">refresh</button>
33+ <label><input type="checkbox" id="auto" checked /> auto (5 s)</label>
34+ <span id="status" class="dim"></span>
35+</header>
36+<main>
37+ <section class="c12"><h2>Session <span id="goal" class="dim"></span></h2><div class="body kpi" id="kpi"></div></section>
38+ <section class="c6"><h2>Agent decisions <span class="dim">why did it do that?</span></h2><div class="body" id="decisions"></div></section>
39+ <section class="c6"><h2>Current page state <span class="dim">semantic DOM</span></h2><div class="body" id="page"></div></section>
40+ <section class="c6"><h2>Visible / recent entities <span id="entcount" class="dim"></span></h2><div class="body" id="entities"></div></section>
41+ <section class="c6"><h2>Detected network schemas <span class="dim">runtime API discovery</span></h2><div class="body" id="schemas"></div></section>
42+ <section class="c4"><h2>Media queue</h2><div class="body" id="media"></div></section>
43+ <section class="c4"><h2>Connector confidence <span class="dim">learned platform model</span></h2><div class="body" id="model"></div></section>
44+ <section class="c4"><h2>World model</h2><div class="body" id="world"></div></section>
45+ <section class="c12"><h2>Recent observations <span class="dim">raw event stream</span></h2><div class="body" id="events"></div></section>
46+</main>
47+<script>
48+const $ = (id) => document.getElementById(id);
49+const esc = (s) => String(s ?? "").replace(/[&<>"]/g, (c) => ({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;" }[c]));
50+const api = (p) => fetch("/api" + p).then((r) => r.json());
51+let current = null;
52+
53+async function loadSessions() {
54+ const sessions = await api("/sessions");
55+ const sel = $("session");
56+ const prev = sel.value;
57+ sel.innerHTML = sessions.map((s) => `<option value="${esc(s.session_id)}">${esc(s.session_id)} · ${esc(s.platform)} · ${esc(s.mode ?? "")} · ${esc(s.health ?? "")}</option>`).join("");
58+ if (prev && sessions.some((s) => s.session_id === prev)) sel.value = prev;
59+ current = sel.value || null;
60+}
61+
62+async function render() {
63+ if (!current) return;
64+ $("status").textContent = "loading…";
65+ const [summary, actions, entities, schemas, media, events, world] = await Promise.all([
66+ api(`/sessions/${current}/summary`), api(`/sessions/${current}/actions`), api(`/sessions/${current}/entities`),
67+ api(`/sessions/${current}/schemas`), api(`/sessions/${current}/media`), api(`/sessions/${current}/events?limit=120`), api(`/sessions/${current}/world`),
68+ ]);
69+ const job = summary.job ?? {}; const sum = summary.summary;
70+ $("goal").textContent = `${job.platform ?? ""} · ${job.mode ?? ""} · ${job.goal ?? ""}`;
71+ const pages = events.filter((e) => e.event_type === "PAGE_OPENED");
72+ const last = pages[0];
73+ $("kpi").innerHTML = [
74+ ["steps", actions.length], ["entities", entities.length], ["schemas", schemas.length], ["media", media.length],
75+ ["events", events.length + (events.length >= 120 ? "+" : "")], ["status", sum ? `ended: ${sum.ended_because}` : "running"],
76+ ["planner", actions[actions.length - 1]?.planner ?? "—"],
77+ ].map(([k, v]) => `<div><b>${esc(v)}</b>${k}</div>`).join("");
78+
79+ $("decisions").innerHTML = `<table><tr><th>#</th><th>action</th><th>gain</th><th>nov</th><th>rel</th><th>planner</th><th>reason</th></tr>` +
80+ actions.slice().reverse().slice(0, 40).map((a) => {
81+ const act = a.action ?? { type: a.action_type, label: a.label };
82+ const gain = a.expected_information_gain ?? a.expected_gain ?? 0;
83+ return `<tr><td>${a.step}</td><td><span class="tag">${esc(act.type)}</span>${esc((act.label || "").slice(0, 70))}</td><td>${Number(gain).toFixed(3)}</td><td>${Number(a.novelty ?? 0).toFixed(2)}</td><td>${Number(a.relevance ?? 0).toFixed(2)}</td><td>${esc(a.planner)}</td><td class="dim">${esc(a.reason)}</td></tr>`;
84+ }).join("") + `</table>`;
85+
86+ if (last) {
87+ const p = last.payload;
88+ $("page").innerHTML = `<div><span class="tag">${esc(p.page_type)}</span> conf ${Number(p.confidence).toFixed(2)} · <a href="${esc(p.url)}" target="_blank">${esc((p.url || "").slice(0, 90))}</a></div>
89+ <div class="dim">entities ${p.entities} · dom ${p.dom_entities} · network ${p.network_entities} · both ${p.both_surfaces} · field agreements ${p.field_agreements} · conflicts ${(p.field_conflicts || []).length} · videos ${p.videos}</div>
90+ <div class="dim">signals: ${esc((p.signals || []).join(", "))}</div><hr style="border-color:var(--line)"/>${esc(p.summary)}`;
91+ } else $("page").textContent = "no page yet";
92+
93+ $("entcount").textContent = `${entities.length}`;
94+ $("entities").innerHTML = `<table><tr><th>type</th><th>name</th><th>author</th><th>metrics</th><th>sources</th><th>conf</th></tr>` +
95+ entities.slice(0, 80).map((e) => {
96+ const prov = e.provenance || (e.fields ? Object.values(e.fields).flatMap((f) => f.provenance || []) : []);
97+ const surfaces = [...new Set(prov.map((p) => p.surface))];
98+ const conf = e.confidence ?? Math.max(0, ...prov.map((p) => p.confidence));
99+ return `<tr><td><span class="tag">${esc(e.type ?? e.entity_type)}</span></td><td>${e.url ? `<a href="${esc(e.url)}" target="_blank">` : ""}${esc((e.name || e.text || e.text_excerpt || e.platform_id || "").slice(0, 80))}${e.url ? "</a>" : ""}</td><td class="dim">${esc((e.author || "").slice(0, 30))}</td><td class="dim">${esc(JSON.stringify(e.metrics || {}))}</td><td>${surfaces.map((s) => `<span class="tag">${esc(s)}</span>`).join("")}</td><td>${Number(conf).toFixed(2)}</td></tr>`;
100+ }).join("") + `</table>`;
101+
102+ $("schemas").innerHTML = `<table><tr><th>endpoint</th><th>shape</th><th>entities</th><th>seen</th></tr>` +
103+ schemas.slice(0, 60).map((s) => {
104+ const fp = s.fingerprint || {}; const sc = s.schema || {};
105+ const types = (sc.candidate_entity_types || []).map((c) => `${c.type} ${Number(c.confidence).toFixed(2)}`).join(", ");
106+ return `<tr><td>${esc(fp.method)} ${esc(fp.hostname)}<span class="dim">${esc(fp.path_pattern)}</span>${fp.graphql_operation ? ` <span class="tag">${esc(fp.graphql_operation)}</span>` : ""}</td><td class="dim">${esc((s.shape_hash || fp.response_shape_hash || "").slice(0, 10))}</td><td>${esc(types)}</td><td>${esc(s.observed_count ?? "")}</td></tr>`;
107+ }).join("") + `</table>`;
108+
109+ $("media").innerHTML = media.length ? media.slice(0, 30).map((m) => `<div><span class="tag">${esc(m.media_type)}</span>${esc((m.title || m.platform_media_id || "").slice(0, 60))} <span class="dim">${m.duration_s ? Math.round(m.duration_s) + "s" : ""} ${m.width ? m.width + "×" + m.height : ""} ${m.delivery?.kind ?? ""}</span>${(m.frames || []).length ? `<div class="frames">${m.frames.map((f) => `<img src="/media/${esc(f.split("/media/")[1] || "")}" />`).join("")}</div>` : ""}</div>`).join("") : "<span class='dim'>no media yet</span>";
110+
111+ if (job.platform) {
112+ const pm = await api(`/platform-model/${job.platform}`);
113+ if (pm && pm.response_patterns) {
114+ const rps = Object.values(pm.response_patterns).filter((r) => Object.keys(r.likely_entity_types || {}).length).sort((a, b) => b.confidence - a.confidence);
115+ const conf = rps.length ? rps.reduce((s, r) => s + r.confidence, 0) / rps.length : 0;
116+ $("model").innerHTML = `<div class="kpi"><div><b>${Math.round(conf * 100)}%</b>schema confidence</div><div><b>${Object.keys(pm.page_types || {}).length}</b>page types</div><div><b>${rps.length}</b>entity schemas</div><div><b>${(pm.sessions_learned || []).length}</b>sessions</div></div>
117+ ${rps.slice(0, 12).map((r) => `<div style="margin-top:6px"><span class="dim">${esc(r.hostname)}${esc(r.path_pattern)}</span> ${Object.keys(r.likely_entity_types).map((t) => `<span class="tag">${esc(t)}</span>`).join("")}<div class="bar"><i style="width:${Math.round(r.confidence * 100)}%"></i></div></div>`).join("")}`;
118+ } else $("model").textContent = "no learned model yet";
119+ }
120+
121+ const nodes = world.nodes || []; const byType = {};
122+ for (const n of nodes) byType[n.type] = (byType[n.type] || 0) + 1;
123+ $("world").innerHTML = `<div class="kpi"><div><b>${nodes.length}</b>nodes</div><div><b>${(world.edges || []).length}</b>edges</div><div><b>${nodes.filter((n) => n.visited).length}</b>visited</div></div>
124+ <div style="margin-top:8px">${Object.entries(byType).map(([t, n]) => `<span class="tag">${esc(t)} ${n}</span>`).join(" ")}</div>
125+ <div class="dim" style="margin-top:8px">${nodes.filter((n) => (n.surfaces || []).length > 1).length} nodes confirmed by ≥2 surfaces</div>`;
126+
127+ $("events").innerHTML = `<table>` + events.slice(0, 120).map((e) => {
128+ const p = e.payload || {}; let brief = "";
129+ if (e.event_type === "NETWORK_RESPONSE_OBSERVED") brief = `${p.kind} ${p.hostname}${p.path_pattern} · ${p.entity_count} entities · ${p.body_size} B`;
130+ else if (e.event_type === "DOM_CHANGED") brief = `+${p.added} −${p.removed} ${JSON.stringify(p.regions)}${p.modal_opened ? " modal" : ""}`;
131+ else if (/DISCOVERED|OBSERVED/.test(e.event_type) && p.entity) brief = `${p.entity.type}: ${(p.entity.name || p.entity.platform_id || "").slice(0, 70)}`;
132+ else if (e.event_type === "ACTION_PLANNED") brief = `${p.action?.type} · ${p.reason}`;
133+ else brief = JSON.stringify(p).slice(0, 140);
134+ const cls = /FAILED|DEGRADED|LOOP|AUTH|ERROR/.test(e.event_type) ? "bad" : /LEARNED|REPAIRED|SCHEMA/.test(e.event_type) ? "ok" : "";
135+ return `<tr><td class="dim">${esc((e.ts || e.timestamp || "").slice(11, 19))}</td><td>#${e.step ?? 0}</td><td class="${cls}">${esc(e.event_type)}</td><td class="dim">${esc(brief)}</td></tr>`;
136+ }).join("") + `</table>`;
137+ $("status").textContent = `updated ${new Date().toLocaleTimeString()}`;
138+}
139+
140+$("session").addEventListener("change", () => { current = $("session").value; render(); });
141+$("refresh").addEventListener("click", () => loadSessions().then(render));
142+setInterval(() => { if ($("auto").checked) loadSessions().then(render); }, 5000);
143+loadSessions().then(render);
144+</script>
145+</body>
146+</html>
added apps/api/src/server.ts +109 −0
@@ -0,0 +1,109 @@
1+import http from "node:http";
2+import fs from "node:fs";
3+import path from "node:path";
4+import { fileURLToPath } from "node:url";
5+import { createLogger, loadConfig } from "@src/shared";
6+import { JsonlEventLog } from "@src/events";
7+import { PostgresStore } from "@src/storage";
8+
9+/**
10+ * Research/Debug dashboard API (§53, §82). Zero framework: node:http + a static page.
11+ * Reads PostgreSQL when configured, else the JSONL session logs.
12+ */
13+const log = createLogger("dashboard");
14+const cfg = loadConfig();
15+const here = path.dirname(fileURLToPath(import.meta.url));
16+const publicDir = path.join(here, "..", "public");
17+let store: PostgresStore | undefined;
18+try {
19+ if (cfg.databaseUrl) store = await PostgresStore.connect(cfg.databaseUrl);
20+} catch (err) {
21+ log.warn("no database — serving JSONL logs", { err: (err as Error).message });
22+}
23+
24+function json(res: http.ServerResponse, data: unknown, status = 200) {
25+ res.writeHead(status, { "content-type": "application/json; charset=utf-8", "access-control-allow-origin": "*" });
26+ res.end(JSON.stringify(data));
27+}
28+
29+function sessionDirs(): string[] {
30+ return fs.existsSync(cfg.sessionsDir) ? fs.readdirSync(cfg.sessionsDir).filter((d) => fs.existsSync(path.join(cfg.sessionsDir, d, "events.jsonl"))).sort().reverse() : [];
31+}
32+
33+function jsonlSessions() {
34+ return sessionDirs().map((id) => {
35+ const job = readJson(path.join(cfg.sessionsDir, id, "job.json")) as Record<string, unknown> | undefined;
36+ const summary = readJson(path.join(cfg.sessionsDir, id, "summary.json")) as Record<string, unknown> | undefined;
37+ return { session_id: id, platform: job?.platform, mode: job?.mode, goal: job?.goal, account_alias: job?.account_alias, started_at: fs.statSync(path.join(cfg.sessionsDir, id)).birthtime, health: summary ? "stopped" : "running", actions: summary?.steps ?? null, entities: summary?.entities ?? null, ended_because: summary?.ended_because };
38+ });
39+}
40+
41+function readJson(file: string): unknown {
42+ try {
43+ return JSON.parse(fs.readFileSync(file, "utf8"));
44+ } catch {
45+ return undefined;
46+ }
47+}
48+
49+const server = http.createServer(async (req, res) => {
50+ const url = new URL(req.url ?? "/", "http://localhost");
51+ const parts = url.pathname.split("/").filter(Boolean);
52+ try {
53+ if (parts[0] === "api") {
54+ if (parts[1] === "sessions" && !parts[2]) return json(res, store ? await store.listSessions() : jsonlSessions());
55+ if (parts[1] === "sessions" && parts[2]) {
56+ const id = parts[2];
57+ const dir = path.join(cfg.sessionsDir, id);
58+ const sub = parts[3];
59+ if (sub === "events") {
60+ const types = url.searchParams.get("types")?.split(",").filter(Boolean);
61+ const limit = Number(url.searchParams.get("limit") ?? 300);
62+ if (store) return json(res, await store.sessionEvents(id, { types, limit }));
63+ const evs = JsonlEventLog.read(dir).filter((e) => !types?.length || types.includes(e.event_type));
64+ return json(res, evs.slice(-limit).reverse());
65+ }
66+ if (sub === "actions") return json(res, store ? await store.sessionActions(id) : JsonlEventLog.read(dir).filter((e) => e.event_type === "ACTION_PLANNED").map((e) => ({ step: e.step, ...(e.payload as Record<string, unknown>) })));
67+ if (sub === "entities") {
68+ if (store) return json(res, await store.sessionEntities(id));
69+ const seen = new Map<string, unknown>();
70+ for (const e of JsonlEventLog.read(dir)) if (/DISCOVERED|ENTITY_OBSERVED/.test(e.event_type) && (e.payload as { entity?: { fingerprint: string } }).entity) seen.set((e.payload as { entity: { fingerprint: string } }).entity.fingerprint, { ...(e.payload as { entity: object }).entity, last_step: e.step, provenance: e.provenance });
71+ return json(res, [...seen.values()].reverse());
72+ }
73+ if (sub === "media") return json(res, store ? await store.sessionMedia(id) : JsonlEventLog.read(dir).filter((e) => e.event_type === "MEDIA_DISCOVERED").map((e) => e.payload));
74+ if (sub === "world") return json(res, readJson(path.join(dir, "world_model.json")) ?? { nodes: [], edges: [] });
75+ if (sub === "summary") return json(res, { job: readJson(path.join(dir, "job.json")), summary: readJson(path.join(dir, "summary.json")) });
76+ if (sub === "schemas") {
77+ const platform = (readJson(path.join(dir, "job.json")) as { platform?: string } | undefined)?.platform;
78+ if (store) return json(res, await store.schemas(platform));
79+ return json(res, JsonlEventLog.read(dir).filter((e) => e.event_type === "NETWORK_SCHEMA_DISCOVERED").map((e) => e.payload));
80+ }
81+ }
82+ if (parts[1] === "platform-model" && parts[2]) {
83+ const f = path.join(cfg.platformModelDir, parts[2], "platform_model.json");
84+ return json(res, readJson(f) ?? { error: "no model yet" });
85+ }
86+ if (parts[1] === "entity" && parts[2]) return json(res, store ? await store.entity(decodeURIComponent(parts[2])) : { error: "needs database" });
87+ if (parts[1] === "stats") return json(res, store ? await store.entityStats() : []);
88+ return json(res, { error: "not found" }, 404);
89+ }
90+ if (parts[0] === "media") {
91+ // serve sampled frames
92+ const file = path.join(cfg.mediaDir, ...parts.slice(1));
93+ if (!file.startsWith(cfg.mediaDir) || !fs.existsSync(file)) return json(res, { error: "not found" }, 404);
94+ res.writeHead(200, { "content-type": "image/jpeg" });
95+ fs.createReadStream(file).pipe(res);
96+ return;
97+ }
98+ const file = path.join(publicDir, parts.length ? parts.join("/") : "index.html");
99+ if (!file.startsWith(publicDir) || !fs.existsSync(file)) return json(res, { error: "not found" }, 404);
100+ const ext = path.extname(file);
101+ res.writeHead(200, { "content-type": ext === ".html" ? "text/html; charset=utf-8" : ext === ".js" ? "text/javascript" : ext === ".css" ? "text/css" : "application/octet-stream" });
102+ fs.createReadStream(file).pipe(res);
103+ } catch (err) {
104+ log.error("request failed", { url: req.url, err: (err as Error).message });
105+ json(res, { error: (err as Error).message }, 500);
106+ }
107+});
108+
109+server.listen(cfg.dashboardPort, () => log.info(`dashboard → http://localhost:${cfg.dashboardPort} (${store ? "postgres" : "jsonl"})`));
added apps/worker/package.json +19 −0
@@ -0,0 +1,19 @@
1+{
2+ "name": "@src/worker",
3+ "version": "0.1.0",
4+ "private": true,
5+ "type": "module",
6+ "dependencies": {
7+ "@src/shared": "workspace:*",
8+ "@src/events": "workspace:*",
9+ "@src/browser": "workspace:*",
10+ "@src/observers": "workspace:*",
11+ "@src/entities": "workspace:*",
12+ "@src/media": "workspace:*",
13+ "@src/agent": "workspace:*",
14+ "@src/platform-model": "workspace:*",
15+ "@src/storage": "workspace:*",
16+ "@src/connectors": "workspace:*",
17+ "playwright": "^1.63.0"
18+ }
19+}
added apps/worker/src/cli.ts +153 −0
@@ -0,0 +1,153 @@
1+import fs from "node:fs";
2+import path from "node:path";
3+import { DEFAULT_BUDGET, isPlatform, loadConfig, newId, type AgentMode, type CrawlJob, type MediaLevel, type Platform } from "@src/shared";
4+import { interactiveLogin } from "@src/browser";
5+import { JsonlEventLog } from "@src/events";
6+import { PlatformModel, compileConnector } from "@src/platform-model";
7+import { CrawlEngine } from "./engine.ts";
8+
9+/**
10+ * social-runtime CLI (§74):
11+ * pnpm src login <platform> [--account alias]
12+ * pnpm src crawl <platform> --goal "…" [--mode research|observe|topic|profile|learn] [--query "…"] [--seed url] [--minutes 10] [--actions 60] [--media 2] [--headless]
13+ * pnpm src learn <platform> [--minutes 15] (PLATFORM LEARNING MODE + connector compilation)
14+ * pnpm src replay <session_id> [--type EVENT_TYPE] (inspect why the crawler did what it did)
15+ * pnpm src status <platform> (learned model summary)
16+ */
17+const [, , cmd, ...rest] = process.argv;
18+const args = parseArgs(rest);
19+const cfg = loadConfig();
20+
21+function parseArgs(list: string[]): { positional: string[]; flags: Record<string, string | boolean> } {
22+ const positional: string[] = [];
23+ const flags: Record<string, string | boolean> = {};
24+ for (let i = 0; i < list.length; i++) {
25+ const a = list[i]!;
26+ if (a.startsWith("--")) {
27+ const key = a.slice(2);
28+ const next = list[i + 1];
29+ if (next !== undefined && !next.startsWith("--")) {
30+ flags[key] = next;
31+ i++;
32+ } else flags[key] = true;
33+ } else positional.push(a);
34+ }
35+ return { positional, flags };
36+}
37+
38+function platformArg(): Platform {
39+ const p = args.positional[0];
40+ if (!p || !isPlatform(p)) {
41+ console.error(`Usage: ${cmd} <platform> — one of youtube, reddit, facebook, instagram, tiktok, x, linkedin, threads`);
42+ process.exit(2);
43+ }
44+ return p;
45+}
46+
47+function str(k: string, d?: string): string | undefined {
48+ const v = args.flags[k];
49+ return typeof v === "string" ? v : d;
50+}
51+function num(k: string, d: number): number {
52+ const v = args.flags[k];
53+ return typeof v === "string" && !Number.isNaN(Number(v)) ? Number(v) : d;
54+}
55+
56+async function main() {
57+ switch (cmd) {
58+ case "login": {
59+ const platform = platformArg();
60+ await interactiveLogin({ platform, accountAlias: str("account", "research-01")!, profilesDir: cfg.profilesDir, channel: cfg.browserChannel });
61+ return;
62+ }
63+ case "crawl":
64+ case "learn": {
65+ const platform = platformArg();
66+ const isLearn = cmd === "learn";
67+ const mode = (str("mode", isLearn ? "learn" : "research") as AgentMode) ?? "research";
68+ const query = str("query") ?? str("q");
69+ const goal = str("goal") ?? (isLearn ? `Learn how ${platform} surfaces content: page types, entity structures, response schemas, navigation` : query ? `Discover public content about “${query}”` : `Observe the ${platform} feed`);
70+ if (args.flags.headless) cfg.headless = true;
71+ const job: CrawlJob = {
72+ job_id: newId("job"),
73+ platform,
74+ account_alias: str("account", "research-01")!,
75+ mode,
76+ goal,
77+ query,
78+ seed_url: str("seed"),
79+ budget: { ...DEFAULT_BUDGET, max_minutes: num("minutes", isLearn ? 15 : DEFAULT_BUDGET.max_minutes), max_actions: num("actions", isLearn ? 80 : DEFAULT_BUDGET.max_actions), max_profiles: num("profiles", DEFAULT_BUDGET.max_profiles), max_posts: num("posts", DEFAULT_BUDGET.max_posts), max_videos: num("videos", DEFAULT_BUDGET.max_videos), max_depth: num("depth", DEFAULT_BUDGET.max_depth) },
80+ media_level: Math.max(0, Math.min(4, num("media", 1))) as MediaLevel,
81+ stay_on_platform: true,
82+ read_only: true,
83+ };
84+ const engine = new CrawlEngine(cfg, job);
85+ const summary = await engine.run();
86+ printSummary(summary);
87+ return;
88+ }
89+ case "status": {
90+ const platform = platformArg();
91+ const model = new PlatformModel(platform, cfg.platformModelDir);
92+ const s = model.summary();
93+ console.log(renderLearned(s));
94+ if (args.flags.compile) console.log("compiled →", compileConnector(model, path.join(process.cwd(), "connectors", platform)));
95+ return;
96+ }
97+ case "replay": {
98+ const id = args.positional[0];
99+ if (!id) {
100+ const dirs = fs.existsSync(cfg.sessionsDir) ? fs.readdirSync(cfg.sessionsDir).sort() : [];
101+ console.log("sessions:\n" + dirs.map((d) => " " + d).join("\n"));
102+ return;
103+ }
104+ const events = JsonlEventLog.read(path.join(cfg.sessionsDir, id));
105+ const type = str("type");
106+ for (const ev of events) {
107+ if (type && ev.event_type !== type) continue;
108+ if (["ACTION_PLANNED", "ACTION_EXECUTED", "ACTION_FAILED", "PAGE_OPENED", "NETWORK_SCHEMA_DISCOVERED", "CONNECTOR_PATTERN_LEARNED", "CONNECTOR_DEGRADED", "LOOP_DETECTED", "AUTH_REQUIRED", "BUDGET_EXHAUSTED", "MEDIA_DISCOVERED"].includes(ev.event_type) || type) {
109+ const p = ev.payload as Record<string, unknown>;
110+ const brief = ev.event_type === "ACTION_PLANNED" ? `${(p.action as { type: string; label: string }).type} — ${(p.action as { label: string }).label} | gain=${Number(p.expected_information_gain).toFixed(3)} | ${p.planner} | ${p.reason}` : ev.event_type === "PAGE_OPENED" ? `${p.page_type} (${Number(p.confidence ?? 0).toFixed(2)}) ${p.url} · entities=${p.entities} dom=${p.dom_entities} net=${p.network_entities} both=${p.both_surfaces}` : ev.event_type === "NETWORK_SCHEMA_DISCOVERED" ? `${(p.fingerprint as { hostname: string; path_pattern: string }).hostname}${(p.fingerprint as { path_pattern: string }).path_pattern} → ${JSON.stringify((p.schema as { candidate_entity_types: unknown }).candidate_entity_types)}` : JSON.stringify(p).slice(0, 220);
111+ console.log(`${ev.timestamp.slice(11, 19)} #${ev.step ?? 0} ${ev.event_type.padEnd(26)} ${brief}`);
112+ }
113+ }
114+ return;
115+ }
116+ default:
117+ console.log(`social-runtime — Social Runtime Crawler
118+
119+ pnpm src login <platform> [--account alias]
120+ pnpm src crawl <platform> --goal "…" [--query "…"] [--mode research|observe|topic|profile|learn] [--seed url] [--minutes 10] [--actions 60] [--media 0-4] [--headless]
121+ pnpm src learn <platform> [--minutes 15]
122+ pnpm src status <platform> [--compile]
123+ pnpm src replay [session_id] [--type EVENT_TYPE]
124+ pnpm dashboard → http://localhost:${cfg.dashboardPort}
125+`);
126+ }
127+}
128+
129+function printSummary(s: Awaited<ReturnType<CrawlEngine["run"]>>) {
130+ console.log(`
131+Session ${s.session_id} (${s.platform}) — ended: ${s.ended_because}
132+ steps: ${s.steps} entities: ${s.entities} videos: ${s.videos}
133+ network responses: ${s.network_responses} distinct schemas: ${s.schemas} patterns learned: ${s.patterns_learned}
134+ world: ${JSON.stringify(s.world.by_type)} multi-surface: ${s.world.multi_surface}
135+${renderLearned(s.connector)}
136+ log: ${s.session_dir}/events.jsonl
137+`);
138+}
139+
140+function renderLearned(s: ReturnType<PlatformModel["summary"]>): string {
141+ return `Platform ${s.platform} learned (${s.sessions} session${s.sessions === 1 ? "" : "s"})
142+ Page types: ${s.page_types}
143+ Entity types: ${s.entity_types}
144+ Navigation actions: ${s.navigation_actions}
145+ Network schemas: ${s.network_schemas}
146+ Media patterns: ${s.media_patterns}
147+ Confidence: ${s.confidence}%`;
148+}
149+
150+main().catch((err) => {
151+ console.error(err);
152+ process.exit(1);
153+});
added apps/worker/src/engine.ts +320 −0
@@ -0,0 +1,320 @@
1+import fs from "node:fs";
2+import path from "node:path";
3+import {
4+ createLogger,
5+ nowIso,
6+ truncate,
7+ type AgentDecision,
8+ type AppConfig,
9+ type CrawlJob,
10+ type ObservedEntity,
11+ type PageState,
12+ type Platform,
13+} from "@src/shared";
14+import { EventBus, JsonlEventLog } from "@src/events";
15+import { SocialBrowserSession } from "@src/browser";
16+import { NetworkObserver, DomObserver, snapshotDom, classifyPage, type ClassifiedResponse } from "@src/observers";
17+import { mergeSurfaces } from "@src/entities";
18+import { detectVideos, sampleVideoFrames } from "@src/media";
19+import { WorldModel, LoopDetector, HeuristicPlanner, LlmPlanner, createLlmClient, type Planner, type GainContext } from "@src/agent";
20+import { PlatformModel, compileConnector } from "@src/platform-model";
21+import { openStore, type PostgresStore } from "@src/storage";
22+import { getAdapter, entitiesFromDom, buildActions, pageFingerprint, type SocialPlatformAdapter } from "@src/connectors";
23+import { ActionExecutor } from "./executor.ts";
24+
25+const log = createLogger("engine");
26+
27+export interface CrawlSummary {
28+ session_id: string;
29+ platform: Platform;
30+ steps: number;
31+ entities: number;
32+ videos: number;
33+ network_responses: number;
34+ schemas: number;
35+ patterns_learned: number;
36+ ended_because: string;
37+ world: ReturnType<WorldModel["stats"]>;
38+ connector: ReturnType<PlatformModel["summary"]>;
39+ session_dir: string;
40+}
41+
42+/**
43+ * The crawl loop (§2): observe → summarize → plan → act → learn, until the budget or the goal ends it.
44+ * Every subsystem is a separate package; this file only orchestrates.
45+ */
46+export class CrawlEngine {
47+ private readonly bus = new EventBus();
48+ private readonly world = new WorldModel();
49+ private readonly loops = new LoopDetector();
50+ private readonly adapter: SocialPlatformAdapter;
51+ private readonly model: PlatformModel;
52+ private session!: SocialBrowserSession;
53+ private net!: NetworkObserver;
54+ private dom!: DomObserver;
55+ private store?: PostgresStore;
56+ private eventLog!: JsonlEventLog;
57+ private planner!: Planner;
58+ private executor!: ActionExecutor;
59+ private sessionDir!: string;
60+ private step = 0;
61+ private stepsWithoutNew = 0;
62+ private counts = { entities: 0, videos: 0, profiles: 0, posts: 0, videos_seen: 0, profiles_seen: 0, posts_seen: 0, patterns: 0, schemas: 0 };
63+ private degraded = false;
64+
65+ constructor(private readonly cfg: AppConfig, private readonly job: CrawlJob) {
66+ this.adapter = getAdapter(job.platform);
67+ this.model = new PlatformModel(job.platform, cfg.platformModelDir);
68+ }
69+
70+ async run(): Promise<CrawlSummary> {
71+ const { job, cfg } = this;
72+ this.session = new SocialBrowserSession({ platform: job.platform, accountAlias: job.account_alias, profilesDir: cfg.profilesDir, headless: cfg.headless, channel: cfg.browserChannel, bus: this.bus });
73+ this.sessionDir = path.join(cfg.sessionsDir, this.session.id);
74+ this.eventLog = new JsonlEventLog(this.sessionDir);
75+ this.eventLog.attach(this.bus);
76+ this.store = await openStore(cfg.databaseUrl);
77+ this.store?.attach(this.bus);
78+ fs.writeFileSync(path.join(this.sessionDir, "job.json"), JSON.stringify({ ...job, session_id: this.session.id }, null, 2));
79+
80+ if (!this.session.hasProfile()) log.warn(`No saved browser profile for ${job.platform}/${job.account_alias}. Run: pnpm login ${job.platform} --account ${job.account_alias}`);
81+
82+ const llm = createLlmClient(cfg);
83+ this.planner = llm ? new LlmPlanner(llm, new HeuristicPlanner(), job.budget.max_llm_tokens) : new HeuristicPlanner();
84+ log.info("planner", { planner: this.planner.name, mode: job.mode, goal: job.goal });
85+
86+ await this.session.start();
87+ // SESSION_STARTED is emitted by the session; enrich the DB row with mode/goal.
88+ this.bus.emit({ event_type: "PAGE_OPENED", platform: job.platform, session_id: this.session.id, step: 0, payload: { url: "about:blank", mode: job.mode, goal: job.goal, budget: job.budget } });
89+ const page = this.session.getPage();
90+ this.net = new NetworkObserver({ page, bus: this.bus, sessionId: this.session.id, platform: job.platform, hints: this.adapter.classifierHints });
91+ this.net.attach();
92+ this.dom = new DomObserver({ page, bus: this.bus, sessionId: this.session.id, platform: job.platform });
93+ await this.dom.attach();
94+ this.executor = new ActionExecutor(page, this.adapter, { stayOnPlatform: job.stay_on_platform });
95+
96+ const started = Date.now();
97+ let endedBecause = "unknown";
98+ try {
99+ // Seed navigation
100+ this.net.beginStep(0, "seed");
101+ this.dom.beginStep(0);
102+ const seed = job.seed_url ?? (job.query && job.mode !== "observe" ? this.adapter.searchUrl(job.query) : this.adapter.homeUrl);
103+ await this.session.navigate(seed);
104+ await page.waitForLoadState("networkidle", { timeout: 6000 }).catch(() => {});
105+ await this.adapter.dismissOverlays?.(page).catch(() => {});
106+ let state = await this.observe(await this.net.collectStep(), "seed");
107+ if (state.classification.page_type === "LOGIN") {
108+ this.session.setHealth("auth_required");
109+ this.bus.emit({ event_type: "AUTH_REQUIRED", platform: job.platform, session_id: this.session.id, step: 0, payload: { url: state.url, hint: `pnpm login ${job.platform} --account ${job.account_alias}` } });
110+ endedBecause = "auth_required";
111+ return this.finish(endedBecause, started);
112+ }
113+
114+ while (true) {
115+ const elapsedMin = (Date.now() - started) / 60_000;
116+ const b = job.budget;
117+ if (this.step >= b.max_actions) { endedBecause = "max_actions"; break; }
118+ if (elapsedMin >= b.max_minutes) { endedBecause = "max_minutes"; break; }
119+ if (this.counts.profiles >= b.max_profiles) { endedBecause = "max_profiles"; break; }
120+ if (this.counts.posts >= b.max_posts) { endedBecause = "max_posts"; break; }
121+ if (this.counts.videos >= b.max_videos) { endedBecause = "max_videos"; break; }
122+ if (this.session.getInfo().health === "crashed") { endedBecause = "browser_crashed"; break; }
123+
124+ this.step++;
125+ const decision = await this.plan(state);
126+ this.bus.emit({ event_type: "ACTION_PLANNED", platform: job.platform, session_id: this.session.id, step: this.step, payload: { action: decision.chosen_action, expected_information_gain: decision.expected_information_gain, novelty: decision.novelty, relevance: decision.relevance, reason: decision.reason, planner: decision.planner, top_scores: decision.scores.slice(0, 5) } });
127+ log.info(`step ${this.step}: ${decision.chosen_action.type} — ${truncate(decision.chosen_action.label, 80)}`, { gain: decision.expected_information_gain.toFixed(3), planner: decision.planner, reason: truncate(decision.reason, 100) });
128+ if (decision.chosen_action.type === "END_SESSION") { endedBecause = "agent_ended"; break; }
129+
130+ const before = { url: state.url, page_type: state.classification.page_type, visible_entities: state.entities.length, world_nodes: this.world.nodes.size };
131+ this.net.beginStep(this.step, decision.chosen_action.id);
132+ this.dom.beginStep(this.step);
133+ const target = decision.chosen_action.target_ref ? state.entities.find((e) => e.ref === decision.chosen_action.target_ref) : undefined;
134+ const result = await this.executor.execute(decision.chosen_action, { entityUrl: target?.url });
135+ this.session.recordAction(decision.chosen_action.type, result.navigated && /^OPEN_/.test(decision.chosen_action.type) ? 1 : decision.chosen_action.type === "BACK" ? -1 : 0);
136+ if (target) this.world.markVisited(target.fingerprint, target.url);
137+ if (result.navigated) this.bus.emit({ event_type: "NAVIGATION_COMPLETED", platform: job.platform, session_id: this.session.id, step: this.step, payload: { from: result.url_before, to: result.url_after, action: decision.chosen_action.type, duration_ms: result.duration_ms } });
138+
139+ const responses = await this.net.collectStep();
140+ const domDeltas = this.dom.collectStep();
141+ const prevState = state;
142+ state = await this.observe(responses, decision.chosen_action.id, target?.fingerprint);
143+ const newEntities = this.world.observe(state.entities, this.step, target?.fingerprint);
144+ this.stepsWithoutNew = newEntities.length ? 0 : this.stepsWithoutNew + 1;
145+ this.tally(newEntities);
146+ if (result.ok && result.navigated) this.tallyVisit(decision.chosen_action.type);
147+
148+ const after = { url: state.url, page_type: state.classification.page_type, visible_entities: state.entities.length, new_entities: newEntities.length, network_responses: responses.length, dom_nodes_added: domDeltas.reduce((s, d) => s + d.added, 0), world_nodes: this.world.nodes.size };
149+ this.bus.emit({ event_type: result.ok ? "ACTION_EXECUTED" : "ACTION_FAILED", platform: job.platform, session_id: this.session.id, step: this.step, payload: { action: decision.chosen_action, ok: result.ok, error: result.error, before, after, duration_ms: result.duration_ms }, discovered_via: { action_id: decision.chosen_action.id } });
150+ await this.store?.recordAction(this.session.id, decision, before, after, result.ok, result.error, result.duration_ms).catch((e) => log.debug("recordAction failed", { err: (e as Error).message }));
151+ await this.persistStep(state, newEntities, responses);
152+
153+ // Learning (§30, §71)
154+ const outcome = this.model.learnStep({ step: this.step, action_type: decision.chosen_action.type, action_id: decision.chosen_action.id, from_page_type: prevState.classification.page_type, to_page_type: state.classification.page_type, to_url: state.url, page_confidence: state.classification.confidence, entities_total: state.entities.length, new_entities: newEntities.length, entity_types: countTypes(state.entities), responses: responses.map((r) => ({ fingerprint: r.fingerprint, entity_count: r.entities.length, kind: r.kind, url: r.response.url, fields: r.schema?.fields.filter((f) => f.semantic[0]?.kind !== "unknown").slice(0, 60).map((f) => ({ path: f.path, semantic: f.semantic[0], example: f.examples[0] })) })), failed: !result.ok }, this.session.id);
155+ if (outcome.newPatterns.length) {
156+ this.counts.patterns += outcome.newPatterns.length;
157+ this.model.announce(this.bus, this.session.id, this.step, outcome.newPatterns);
158+ }
159+ this.model.save();
160+
161+ // Degradation / self-healing signal (§32)
162+ const degradedNow = this.model.isDegraded(state.classification.page_type, state.entities.length, this.adapter.expectedEntities(state.classification.page_type));
163+ if (degradedNow && !this.degraded) {
164+ this.degraded = true;
165+ this.bus.emit({ event_type: "CONNECTOR_DEGRADED", platform: job.platform, session_id: this.session.id, step: this.step, payload: { page_type: state.classification.page_type, expected: this.adapter.expectedEntities(state.classification.page_type), observed: 0, url: state.url, action: "switching to exploration heuristics" } });
166+ } else if (!degradedNow && this.degraded && state.entities.length > 0) {
167+ this.degraded = false;
168+ this.bus.emit({ event_type: "CONNECTOR_REPAIRED", platform: job.platform, session_id: this.session.id, step: this.step, payload: { page_type: state.classification.page_type, observed: state.entities.length } });
169+ }
170+
171+ // Loop safety (§41)
172+ this.loops.record(state.fingerprint, `${decision.chosen_action.type}:${decision.chosen_action.target_url ?? ""}`);
173+ const loop = this.loops.detect();
174+ if (loop.loop) {
175+ this.bus.emit({ event_type: "LOOP_DETECTED", platform: job.platform, session_id: this.session.id, step: this.step, payload: { pattern: loop.pattern, url: state.url } });
176+ log.warn("navigation loop detected — breaking out", { pattern: loop.pattern });
177+ await this.executor.execute({ id: "loopbreak", type: "RETURN_TO_FEED", label: "loop break", cost: 1, target_url: job.query ? this.adapter.searchUrl(job.query) : this.adapter.homeUrl }, {});
178+ state = await this.observe(await this.net.collectStep(), "loopbreak");
179+ }
180+ if (state.classification.page_type === "LOGIN") {
181+ this.session.setHealth("auth_required");
182+ this.bus.emit({ event_type: "AUTH_REQUIRED", platform: job.platform, session_id: this.session.id, step: this.step, payload: { url: state.url } });
183+ endedBecause = "auth_required";
184+ break;
185+ }
186+ if (this.stepsWithoutNew >= 10) { endedBecause = "no_new_information"; break; }
187+ }
188+ } catch (err) {
189+ endedBecause = `error: ${(err as Error).message.split("\n")[0]}`;
190+ log.error("engine error", { err: (err as Error).stack?.split("\n").slice(0, 3).join(" | ") });
191+ this.bus.emit({ event_type: "WORKER_ERROR", platform: job.platform, session_id: this.session.id, step: this.step, payload: { error: (err as Error).message } });
192+ }
193+ return this.finish(endedBecause, started);
194+ }
195+
196+ /** Observe the current page across surfaces and build the compact PageState (§9, §14). */
197+ private async observe(responses: ClassifiedResponse[], via: string, discoveredFrom?: string): Promise<PageState> {
198+ const page = this.session.getPage();
199+ const snapshot = await snapshotDom(page).catch((err: Error) => {
200+ log.warn("dom snapshot failed", { err: err.message.split("\n")[0] });
201+ return undefined;
202+ });
203+ if (!snapshot) {
204+ return { url: page.url(), title: "", platform: this.job.platform, classification: { page_type: "UNKNOWN", confidence: 0.1, signals: ["snapshot failed"] }, entities: [], actions: [{ id: "A1", type: "WAIT_FOR_CONTENT", label: "Wait for content", cost: 1 }, { id: "A2", type: "RETURN_TO_FEED", label: "Return to feed", cost: 2, target_url: this.adapter.homeUrl }, { id: "A3", type: "END_SESSION", label: "End", cost: 0.5 }], media: [], summary_text: "", fingerprint: pageFingerprint(page.url(), []), captured_at: nowIso() };
205+ }
206+ const classification = classifyPage(snapshot, this.adapter.pageTypeHints);
207+ const domEntities = entitiesFromDom(snapshot, this.adapter);
208+ const netEntities = responses.flatMap((r) => r.entities);
209+ const merged = mergeSurfaces(netEntities, domEntities);
210+ // Keep the planner's list focused: DOM-visible entities first, then network-only ones that have a page URL.
211+ const entities = merged.merged.filter((e) => e.provenance.some((p) => p.surface === "dom") || e.url).slice(0, 80);
212+ const media = detectVideos({ platform: this.job.platform, snapshot, entities, responses, pageUrl: snapshot.url });
213+ const visited = new Set([...this.world.nodes.values()].filter((n) => n.visited).map((n) => n.fingerprint));
214+ const actions = buildActions(entities, snapshot, this.adapter, { query: this.job.query, visited });
215+ const summary = renderSummary(snapshot.title, classification.page_type, entities, actions);
216+ const state: PageState = { url: snapshot.url, title: snapshot.title, platform: this.job.platform, classification, entities, actions, media, summary_text: summary, fingerprint: pageFingerprint(snapshot.url, entities), captured_at: nowIso() };
217+
218+ this.bus.emit({ event_type: "PAGE_OPENED", platform: this.job.platform, session_id: this.session.id, step: this.step, payload: { url: snapshot.url, title: snapshot.title, page_type: classification.page_type, confidence: classification.confidence, signals: classification.signals, entities: entities.length, dom_entities: domEntities.length, network_entities: netEntities.length, both_surfaces: merged.both, field_agreements: merged.field_agreements, field_conflicts: merged.field_conflicts.slice(0, 5), videos: media.length, actions: actions.length, summary }, provenance: [{ surface: "dom", confidence: classification.confidence }], discovered_via: { action_id: via } });
219+ this.bus.emit({ event_type: "PAGE_CLASSIFIED", platform: this.job.platform, session_id: this.session.id, step: this.step, payload: { url: snapshot.url, ...classification } });
220+ for (const e of entities) {
221+ const type = e.type === "video" ? "VIDEO_DISCOVERED" : e.type === "post" ? "POST_DISCOVERED" : e.type === "comment" ? "COMMENT_DISCOVERED" : e.type === "profile" || e.type === "channel" || e.type === "person" ? "PROFILE_DISCOVERED" : "ENTITY_OBSERVED";
222+ const known = this.world.nodes.has(e.fingerprint);
223+ this.bus.emit({ event_type: known ? "ENTITY_OBSERVED" : type, platform: this.job.platform, session_id: this.session.id, step: this.step, payload: { entity: { type: e.type, platform_id: e.platform_id, url: e.url, name: e.name, text: e.text, author: e.author, metrics: e.metrics, media: e.media, context: e.context, fingerprint: e.fingerprint, canonical_id: null }, fields: e.fields }, provenance: e.provenance, discovered_via: { action_id: via, url: snapshot.url } });
224+ }
225+ for (const m of media) if (m.width || m.duration_s || m.delivery) this.bus.emit({ event_type: "MEDIA_DISCOVERED", platform: this.job.platform, session_id: this.session.id, step: this.step, payload: { ...m } as Record<string, unknown>, provenance: m.provenance, discovered_via: { action_id: via } });
226+ if (this.step === 0) this.world.observe(entities, 0, discoveredFrom);
227+ return state;
228+ }
229+
230+ private async plan(state: PageState): Promise<AgentDecision> {
231+ // Relevance is measured against the goal *and* the topic query (the query carries the domain words).
232+ const goal = this.job.query ? `${this.job.goal} ${this.job.query}` : this.job.goal;
233+ const ctx: GainContext = { goal, mode: this.job.mode, world: this.world, state, recentActionTypes: this.loops.recentActionTypes(), recentUrls: [...this.world.visitedUrls].slice(-10), stepsWithoutNewEntities: this.stepsWithoutNew, learnedYield: this.model.learnedYield() };
234+ return this.planner.plan(ctx, this.step);
235+ }
236+
237+ /** Discovery counters (what we have seen). */
238+ private tally(fresh: ObservedEntity[]): void {
239+ for (const e of fresh) {
240+ this.counts.entities++;
241+ if (e.type === "video") this.counts.videos_seen++;
242+ else if (e.type === "post" || e.type === "comment") this.counts.posts_seen++;
243+ else if (e.type === "profile" || e.type === "channel" || e.type === "person" || e.type === "page") this.counts.profiles_seen++;
244+ }
245+ }
246+
247+ /** Budget counters (§40) count pages we actually opened, not entities merely seen in a feed. */
248+ private tallyVisit(actionType: string): void {
249+ if (/^OPEN_(CHANNEL|PROFILE|PAGE)$/.test(actionType)) this.counts.profiles++;
250+ else if (/^OPEN_(POST|COMMENTS|ENTITY)$/.test(actionType)) this.counts.posts++;
251+ else if (actionType === "OPEN_VIDEO") this.counts.videos++;
252+ }
253+
254+ private async persistStep(state: PageState, fresh: ObservedEntity[], responses: ClassifiedResponse[]): Promise<void> {
255+ const ts = nowIso();
256+ const frames: Record<string, string[]> = {};
257+ if (this.job.media_level >= 2) {
258+ const current = state.media.find((m) => m.width || m.delivery?.hostnames.length);
259+ if (current) {
260+ const files = await sampleVideoFrames(this.session.getPage(), current, this.cfg.mediaDir, this.job.media_level).catch(() => []);
261+ if (files.length) {
262+ frames[current.fingerprint] = files;
263+ this.bus.emit({ event_type: "VIDEO_DISCOVERED", platform: this.job.platform, session_id: this.session.id, step: this.step, payload: { fingerprint: current.fingerprint, title: current.title, frames: files, level: this.job.media_level }, provenance: [{ surface: "visual", confidence: 0.8 }] });
264+ }
265+ }
266+ }
267+ if (!this.store) return;
268+ try {
269+ await this.store.upsertEntities(state.entities, this.session.id, this.step, ts);
270+ await this.store.upsertMedia(state.media, frames, ts);
271+ await this.store.recordFeedItems(this.session.id, this.step, state, ts);
272+ const recentEdges = this.world.edges.filter((e) => e.step === this.step);
273+ await this.store.recordRelationships(this.session.id, recentEdges);
274+ } catch (err) {
275+ log.debug("persist failed", { err: (err as Error).message, fresh: fresh.length, responses: responses.length });
276+ }
277+ }
278+
279+ private async finish(endedBecause: string, started: number): Promise<CrawlSummary> {
280+ this.model.save();
281+ const connector = this.model.summary();
282+ if (connector.confidence >= 40) {
283+ const manifest = compileConnector(this.model, path.join(process.cwd(), "connectors", this.job.platform));
284+ await this.store?.recordConnectorVersion(this.job.platform, connector, manifest).catch(() => {});
285+ }
286+ const world = this.world.stats();
287+ fs.writeFileSync(path.join(this.sessionDir, "world_model.json"), JSON.stringify(this.world.toJSON(), null, 2));
288+ const summary: CrawlSummary = { session_id: this.session.id, platform: this.job.platform, steps: this.step, entities: this.world.nodes.size, videos: this.counts.videos_seen, network_responses: this.net?.totalResponses ?? 0, schemas: this.net?.seenShapeHashes.size ?? 0, patterns_learned: this.counts.patterns, ended_because: endedBecause, world, connector, session_dir: this.sessionDir };
289+ this.bus.emit({ event_type: "BUDGET_EXHAUSTED", platform: this.job.platform, session_id: this.session.id, step: this.step, payload: { reason: endedBecause, elapsed_ms: Date.now() - started } });
290+ await this.session.stop();
291+ await this.bus.flush();
292+ fs.writeFileSync(path.join(this.sessionDir, "summary.json"), JSON.stringify(summary, null, 2));
293+ await this.eventLog.close();
294+ await this.store?.close();
295+ return summary;
296+ }
297+}
298+
299+function countTypes(entities: ObservedEntity[]): Record<string, number> {
300+ const out: Record<string, number> = {};
301+ for (const e of entities) out[e.type] = (out[e.type] ?? 0) + 1;
302+ return out;
303+}
304+
305+/** Textual semantic page representation (§14) — what a planner (or a human in the dashboard) reads. */
306+export function renderSummary(title: string, pageType: string, entities: ObservedEntity[], actions: PageState["actions"]): string {
307+ const lines = [`PAGE: ${pageType} — ${truncate(title, 90)}`, "", "VISIBLE ENTITIES"];
308+ for (const e of entities.slice(0, 25)) {
309+ lines.push(`[${e.ref}] ${e.type}${e.context ? ` (${e.context})` : ""}`);
310+ if (e.name) lines.push(` Name: ${truncate(e.name, 100)}`);
311+ if (e.author) lines.push(` Author: ${truncate(e.author, 60)}`);
312+ if (e.media?.has_video) lines.push(` Video: yes${e.media.duration_s ? ` (${Math.round(e.media.duration_s)}s)` : ""}`);
313+ if (e.metrics && Object.keys(e.metrics).length) lines.push(` Metrics: ${Object.entries(e.metrics).map(([k, v]) => `${k}=${v}`).join(", ")}`);
314+ lines.push(` Sources: ${[...new Set(e.provenance.map((p) => p.surface))].join("+")}`);
315+ }
316+ if (entities.length > 25) lines.push(`… ${entities.length - 25} more`);
317+ lines.push("", "ACTIONS");
318+ for (const a of actions) lines.push(`[${a.id}] ${a.label}`);
319+ return lines.join("\n");
320+}
added apps/worker/src/executor.ts +0 −0

Binary file not shown.

added connectors/youtube/manifest.json +165 −0
@@ -0,0 +1,165 @@
1+{
2+ "platform": "youtube",
3+ "compiled_at": "2026-09-11T21:32:58.338Z",
4+ "learned_from_sessions": 4,
5+ "confidence": 45,
6+ "page_types": [
7+ {
8+ "page_type": "UNKNOWN",
9+ "url_patterns": [
10+ "/",
11+ "/results"
12+ ],
13+ "avg_entities": 0
14+ },
15+ {
16+ "page_type": "CHANNEL",
17+ "url_patterns": [
18+ "/@Mila-Quebec-AI-Institute"
19+ ],
20+ "avg_entities": 16
21+ },
22+ {
23+ "page_type": "SEARCH_RESULTS",
24+ "url_patterns": [
25+ "/results"
26+ ],
27+ "avg_entities": 29
28+ },
29+ {
30+ "page_type": "VIDEO_DETAIL",
31+ "url_patterns": [
32+ "/watch"
33+ ],
34+ "avg_entities": 39
35+ }
36+ ],
37+ "network_patterns": [],
38+ "navigation": [
39+ {
40+ "action": "WAIT_FOR_CONTENT",
41+ "from": "UNKNOWN",
42+ "to": {
43+ "UNKNOWN": 5
44+ },
45+ "avg_new_entities": 0,
46+ "observed": 5
47+ },
48+ {
49+ "action": "RETURN_TO_FEED",
50+ "from": "UNKNOWN",
51+ "to": {
52+ "UNKNOWN": 2
53+ },
54+ "avg_new_entities": 0,
55+ "observed": 2
56+ },
57+ {
58+ "action": "OPEN_CHANNEL",
59+ "from": "SEARCH_RESULTS",
60+ "to": {
61+ "CHANNEL": 1
62+ },
63+ "avg_new_entities": 14,
64+ "observed": 1
65+ },
66+ {
67+ "action": "SEARCH",
68+ "from": "CHANNEL",
69+ "to": {
70+ "SEARCH_RESULTS": 1
71+ },
72+ "avg_new_entities": 1,
73+ "observed": 1
74+ },
75+ {
76+ "action": "OPEN_VIDEO",
77+ "from": "SEARCH_RESULTS",
78+ "to": {
79+ "VIDEO_DETAIL": 3
80+ },
81+ "avg_new_entities": 25.7,
82+ "observed": 3
83+ },
84+ {
85+ "action": "OPEN_VIDEO",
86+ "from": "VIDEO_DETAIL",
87+ "to": {
88+ "VIDEO_DETAIL": 10
89+ },
90+ "avg_new_entities": 29.3,
91+ "observed": 10
92+ },
93+ {
94+ "action": "SEARCH",
95+ "from": "VIDEO_DETAIL",
96+ "to": {
97+ "SEARCH_RESULTS": 1
98+ },
99+ "avg_new_entities": 1,
100+ "observed": 1
101+ }
102+ ],
103+ "media": [
104+ {
105+ "hostname": "www.youtube.com",
106+ "kind": "media_segment",
107+ "observed_count": 12
108+ },
109+ {
110+ "hostname": "rr4---sn-q4fl6nde.googlevideo.com",
111+ "kind": "media_segment",
112+ "observed_count": 6
113+ },
114+ {
115+ "hostname": "i.ytimg.com",
116+ "kind": "image",
117+ "observed_count": 172
118+ },
119+ {
120+ "hostname": "rr5---sn-q4fl6n6z.googlevideo.com",
121+ "kind": "media_segment",
122+ "observed_count": 6
123+ },
124+ {
125+ "hostname": "rr1---sn-cxaaj5o5q5-t0ar.googlevideo.com",
126+ "kind": "media_segment",
127+ "observed_count": 11
128+ },
129+ {
130+ "hostname": "rr2---sn-cxaaj5o5q5-t0ay.googlevideo.com",
131+ "kind": "media_segment",
132+ "observed_count": 2
133+ },
134+ {
135+ "hostname": "rr3---sn-cxaaj5o5q5-t0ay.googlevideo.com",
136+ "kind": "media_segment",
137+ "observed_count": 1
138+ },
139+ {
140+ "hostname": "rr1---sn-cxaaj5o5q5-t0a6.googlevideo.com",
141+ "kind": "media_segment",
142+ "observed_count": 1
143+ },
144+ {
145+ "hostname": "rr5---sn-cxaaj5o5q5-t0ay.googlevideo.com",
146+ "kind": "media_segment",
147+ "observed_count": 1
148+ },
149+ {
150+ "hostname": "rr4---sn-cxaaj5o5q5-t0ay.googlevideo.com",
151+ "kind": "media_segment",
152+ "observed_count": 2
153+ },
154+ {
155+ "hostname": "rr1---sn-cxaaj5o5q5-t0ay.googlevideo.com",
156+ "kind": "media_segment",
157+ "observed_count": 3
158+ },
159+ {
160+ "hostname": "rr1---sn-cxaaj5o5q5-t0ad.googlevideo.com",
161+ "kind": "media_segment",
162+ "observed_count": 1
163+ }
164+ ]
165+}
\ No newline at end of file
added data/platform_model/.gitkeep +0 −0
added docs/ARCHITECTURE.md +51 −0
@@ -0,0 +1,51 @@
1+# Architecture notes — v0.1
2+
3+## The loop (apps/worker/src/engine.ts)
4+
5+```
6+seed (search url / home) NetworkObserver.beginStep / DomObserver.beginStep
7+ ↓
8+observe() = snapshotDom → classifyPage → entitiesFromDom ⊕ mineEntities(network) → mergeSurfaces → detectVideos → buildActions → PageState
9+ ↓
10+plan() = scoreActions (information gain) → HeuristicPlanner | LlmPlanner (only when unsure)
11+ ↓
12+execute() = ActionExecutor (click link / scroll / search / expand / play / back …) ← the only place actions touch the browser
13+ ↓
14+collect network + DOM deltas of the step → observe() again → WorldModel.observe (fresh entities)
15+ ↓
16+PlatformModel.learnStep (action → responses → entities) → CONNECTOR_PATTERN_LEARNED / CONNECTOR_DEGRADED
17+ ↓
18+LoopDetector, budget, AUTH_REQUIRED checks → next step
19+```
20+
21+Everything is published on the `EventBus` (Universal Social Event Format) and consumed by the JSONL log (replay) and the PostgreSQL store.
22+
23+## Surfaces and provenance
24+
25+Every `ObservedEntity` carries `provenance[]` and `fields{name → {value, provenance[]}}`. The merge (`packages/entities`) keeps both network and DOM evidence, counts field agreements and conflicts, and boosts confidence when two surfaces agree. `PAGE_OPENED` events record `dom_entities / network_entities / both_surfaces / field_agreements` so the "DOM vs network" experiment (§69) is measurable from the logs.
26+
27+## Generic vs platform-specific
28+
29+Generic: browser lifecycle, network capture, `SchemaProfiler`, JSON entity miner, DOM snapshot, page classifier, planner, information gain, world model, learning, storage, dashboard.
30+
31+Adapter (`packages/connectors/<platform>.ts`): hosts, home/search URL, URL grammar → entity type + id, page-type URL hints, id key names, rich-text collapsers (`runs`/`simpleText` on YouTube), noise-link filter, expected entity counts per page type, overlay dismissal.
32+
33+Adapters contain **no endpoints and no CSS classes**. The network layer fingerprints endpoints by `(hostname, path pattern, method, response shape hash)`.
34+
35+## Learned state
36+
37+`data/platform_model/<platform>/platform_model.json` (plus split files): response patterns (shape hash → likely entity types, confidence, triggered_by action), action patterns (avg new entities, target page types, usual shapes), page types (url patterns, avg entities), media patterns. `compileConnector()` writes `connectors/<platform>/manifest.json` when confidence ≥ 40 %.
38+
39+`PlatformModel.learnedYield()` feeds the expected-entity-yield term of the information-gain formula, so navigation improves with what the crawler has learned.
40+
41+## Degradation / self-healing (first step)
42+
43+`PlatformModel.isDegraded(pageType, observed, adapterExpected)` fires `CONNECTOR_DEGRADED` when a well-known page type yields zero entities; `CONNECTOR_REPAIRED` when extraction recovers. Full re-learning (re-inspect DOM/network, regenerate the connector, run regression tests) is Phase 12.
44+
45+## Next steps
46+
47+1. Run authenticated sessions on YouTube and Reddit, inspect field conflicts in the dashboard, tune adapters and the miner.
48+2. Embeddings for novelty (local model) behind `WorldModel.novelty`.
49+3. Facebook adapter (Phase 2) through PLATFORM LEARNING mode first: `pnpm learn facebook` once a GenericAdapter fallback exists.
50+4. Record/replay fixtures from `sessions/<id>/events.jsonl` for regression tests of learned schemas.
51+5. Distributed workers on MacLustr nodes (one browser profile per node), central PostgreSQL.
added package.json +29 −0
@@ -0,0 +1,29 @@
1+{
2+ "name": "social-runtime-crawler",
3+ "version": "0.1.0",
4+ "private": true,
5+ "description": "Social Runtime Crawler — browser-native, self-learning crawler that mines the observable runtime of social applications.",
6+ "type": "module",
7+ "packageManager": "pnpm@11.1.2",
8+ "engines": {
9+ "node": ">=22"
10+ },
11+ "scripts": {
12+ "src": "tsx apps/worker/src/cli.ts",
13+ "login": "tsx apps/worker/src/cli.ts login",
14+ "crawl": "tsx apps/worker/src/cli.ts crawl",
15+ "learn": "tsx apps/worker/src/cli.ts learn",
16+ "replay": "tsx apps/worker/src/cli.ts replay",
17+ "dashboard": "tsx apps/api/src/server.ts",
18+ "db:migrate": "tsx packages/storage/src/migrate.ts",
19+ "typecheck": "tsc -p tsconfig.json --noEmit",
20+ "test": "vitest run",
21+ "test:watch": "vitest"
22+ },
23+ "devDependencies": {
24+ "@types/node": "^24.0.0",
25+ "tsx": "^4.20.0",
26+ "typescript": "^5.9.3",
27+ "vitest": "^3.2.0"
28+ }
29+}
added packages/agent/package.json +16 −0
@@ -0,0 +1,16 @@
1+{
2+ "name": "@src/agent",
3+ "version": "0.1.0",
4+ "private": true,
5+ "type": "module",
6+ "main": "./src/index.ts",
7+ "types": "./src/index.ts",
8+ "exports": {
9+ ".": "./src/index.ts"
10+ },
11+ "dependencies": {
12+ "@src/shared": "workspace:*",
13+ "@src/events": "workspace:*",
14+ "@anthropic-ai/sdk": "^0.125.0"
15+ }
16+}
added packages/agent/src/InformationGain.ts +125 −0
@@ -0,0 +1,125 @@
1+import { clamp01, jaccard, tokenize, type ActionScore, type AgentMode, type ObservedEntity, type PageState, type SemanticAction } from "@src/shared";
2+import type { WorldModel } from "./WorldModel.ts";
3+
4+/**
5+ * Information Gain Engine (§26–§27):
6+ * gain = novelty × relevance × expected_entity_yield × confidence × source_quality ÷ cost
7+ * with penalties for already-seen / duplicate / loop / low-quality / low-relevance.
8+ */
9+export interface GainContext {
10+ goal: string;
11+ mode: AgentMode;
12+ world: WorldModel;
13+ state: PageState;
14+ recentActionTypes: string[]; // last N action types
15+ recentUrls: string[];
16+ stepsWithoutNewEntities: number;
17+ /** Learned expectations: action type → average new entities produced (from the platform model). */
18+ learnedYield?: Record<string, number>;
19+}
20+
21+const TYPE_YIELD: Record<string, number> = {
22+ OPEN_CHANNEL: 0.8,
23+ OPEN_PROFILE: 0.8,
24+ OPEN_PAGE: 0.75,
25+ OPEN_VIDEO: 0.65,
26+ OPEN_POST: 0.65,
27+ OPEN_COMMENTS: 0.55,
28+ OPEN_ENTITY: 0.6,
29+ SCROLL_DOWN: 0.6,
30+ SEARCH: 0.85,
31+ EXPAND: 0.4,
32+ PLAY_VIDEO: 0.3,
33+ SCROLL_UP: 0.05,
34+ BACK: 0.2,
35+ RETURN_TO_FEED: 0.3,
36+ FILTER: 0.4,
37+ WAIT_FOR_CONTENT: 0.1,
38+ END_SESSION: 0.0,
39+ FORWARD: 0.1,
40+ PAUSE_VIDEO: 0.0,
41+ COLLAPSE: 0.0,
42+};
43+
44+export function relevanceOf(text: string | undefined, goal: string): number {
45+ if (!goal.trim()) return 0.6;
46+ if (!text) return 0.35;
47+ const g = tokenize(goal);
48+ const t = tokenize(text);
49+ if (g.size === 0 || t.size === 0) return 0.4;
50+ let hits = 0;
51+ for (const tok of g) if (t.has(tok) || [...t].some((x) => x.startsWith(tok.slice(0, 5)) && tok.length > 4)) hits++;
52+ const coverage = hits / g.size;
53+ return clamp01(0.3 + 0.7 * coverage + 0.2 * jaccard(g, t));
54+}
55+
56+function entityOf(ctx: GainContext, a: SemanticAction): ObservedEntity | undefined {
57+ return a.target_ref ? ctx.state.entities.find((e) => e.ref === a.target_ref) : undefined;
58+}
59+
60+export function scoreActions(ctx: GainContext, actions: SemanticAction[]): ActionScore[] {
61+ const scores: ActionScore[] = [];
62+ const currentRel = relevanceOf(ctx.state.summary_text.slice(0, 1500), ctx.goal);
63+ for (const a of actions) {
64+ const penalties: string[] = [];
65+ const e = entityOf(ctx, a);
66+ let novelty = 0.5;
67+ let relevance = 0.5;
68+ let confidence = 0.7;
69+ let source_quality = 0.7;
70+ let yieldExp = ctx.learnedYield?.[a.type] !== undefined ? clamp01(ctx.learnedYield[a.type]! / 10) : TYPE_YIELD[a.type] ?? 0.3;
71+
72+ if (e) {
73+ novelty = ctx.world.novelty(e);
74+ relevance = relevanceOf(`${e.name ?? ""} ${e.text ?? ""} ${e.author ?? ""}`, ctx.goal);
75+ confidence = Math.max(0, ...e.provenance.map((p) => p.confidence));
76+ source_quality = e.provenance.some((p) => p.surface === "network") && e.provenance.some((p) => p.surface === "dom") ? 0.95 : 0.75;
77+ if (ctx.world.isVisited(e)) penalties.push("already_seen");
78+ if (/^(navigation|header|sidebar)/.test(e.context ?? "") && ctx.mode !== "learn") penalties.push("low_quality_source");
79+ if (!e.name && !e.text) penalties.push("low_confidence_entity");
80+ // Profiles/channels are hubs: raise yield when the goal is about people/orgs.
81+ if ((e.type === "channel" || e.type === "profile") && /personalit|people|person|creator|influenc|chaîne|channel|profil/i.test(ctx.goal)) yieldExp = Math.min(1, yieldExp + 0.15);
82+ } else {
83+ switch (a.type) {
84+ case "SCROLL_DOWN":
85+ relevance = currentRel;
86+ novelty = ctx.stepsWithoutNewEntities > 2 ? 0.2 : 0.6;
87+ if (ctx.recentActionTypes.slice(-4).every((t) => t === "SCROLL_DOWN") && ctx.recentActionTypes.length >= 4) penalties.push("navigation_loop");
88+ break;
89+ case "SEARCH":
90+ relevance = 0.9;
91+ novelty = ctx.recentActionTypes.includes("SEARCH") ? 0.15 : 0.9;
92+ if (ctx.state.classification.page_type === "SEARCH_RESULTS") penalties.push("already_seen");
93+ break;
94+ case "EXPAND":
95+ relevance = currentRel;
96+ novelty = 0.5;
97+ break;
98+ case "PLAY_VIDEO":
99+ relevance = currentRel;
100+ novelty = ctx.state.classification.page_type === "VIDEO_DETAIL" ? 0.6 : 0.3;
101+ break;
102+ case "BACK":
103+ case "RETURN_TO_FEED":
104+ relevance = 0.4;
105+ novelty = ctx.stepsWithoutNewEntities > 1 ? 0.6 : 0.2;
106+ if (ctx.recentActionTypes.slice(-2).includes(a.type)) penalties.push("navigation_loop");
107+ break;
108+ case "END_SESSION":
109+ relevance = 0.1;
110+ novelty = ctx.stepsWithoutNewEntities > 6 ? 0.9 : 0.05;
111+ break;
112+ default:
113+ break;
114+ }
115+ }
116+ if (ctx.mode === "observe" && a.type !== "SCROLL_DOWN" && a.type !== "WAIT_FOR_CONTENT" && a.type !== "END_SESSION") penalties.push("mode_observe_no_navigation");
117+ if (relevance < 0.35 && e) penalties.push("low_relevance");
118+
119+ let gain = (novelty * relevance * yieldExp * confidence * source_quality) / Math.max(0.25, a.cost);
120+ for (const p of penalties) gain *= p === "already_seen" ? 0.05 : p === "navigation_loop" ? 0.15 : p === "mode_observe_no_navigation" ? 0.02 : 0.5;
121+
122+ scores.push({ action_id: a.id, novelty, relevance, expected_entity_yield: yieldExp, confidence, source_quality, cost: a.cost, penalties, information_gain: gain });
123+ }
124+ return scores.sort((x, y) => y.information_gain - x.information_gain);
125+}
added packages/agent/src/LoopDetector.ts +32 −0
Large diff (32 lines) is not loaded by default to keep the page fast.Load diff
added packages/agent/src/WorldModel.ts +124 −0
Large diff (124 lines) is not loaded by default to keep the page fast.Load diff
added packages/agent/src/agent.test.ts +69 −0
Large diff (69 lines) is not loaded by default to keep the page fast.Load diff
added packages/agent/src/index.ts +5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added packages/agent/src/llm.ts +78 −0
Large diff (78 lines) is not loaded by default to keep the page fast.Load diff
added packages/agent/src/planner.ts +123 −0
Large diff (123 lines) is not loaded by default to keep the page fast.Load diff
added packages/browser/package.json +16 −0
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
added packages/browser/src/index.ts +2 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added packages/browser/src/login.ts +37 −0
Large diff (37 lines) is not loaded by default to keep the page fast.Load diff
added packages/browser/src/session.ts +131 −0
Large diff (131 lines) is not loaded by default to keep the page fast.Load diff
added packages/connectors/package.json +16 −0
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
added packages/connectors/src/adapter.ts +182 −0
Large diff (182 lines) is not loaded by default to keep the page fast.Load diff
added packages/connectors/src/index.ts +22 −0
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff
added packages/connectors/src/reddit.ts +41 −0
Large diff (41 lines) is not loaded by default to keep the page fast.Load diff
added packages/connectors/src/youtube.ts +86 −0
Large diff (86 lines) is not loaded by default to keep the page fast.Load diff
added packages/entities/package.json +15 −0
Large diff (15 lines) is not loaded by default to keep the page fast.Load diff
added packages/entities/src/entities.test.ts +38 −0
Large diff (38 lines) is not loaded by default to keep the page fast.Load diff
added packages/entities/src/index.ts +135 −0
Large diff (135 lines) is not loaded by default to keep the page fast.Load diff
added packages/events/package.json +14 −0
Large diff (14 lines) is not loaded by default to keep the page fast.Load diff
added packages/events/src/index.ts +128 −0
Large diff (128 lines) is not loaded by default to keep the page fast.Load diff
added packages/media/package.json +16 −0
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
added packages/media/src/index.ts +117 −0
Large diff (117 lines) is not loaded by default to keep the page fast.Load diff
added packages/observers/package.json +16 −0
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
added packages/observers/src/PageClassifier.test.ts +22 −0
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff
added packages/observers/src/PageClassifier.ts +53 −0
Large diff (53 lines) is not loaded by default to keep the page fast.Load diff
added packages/observers/src/dom/DomObserver.ts +134 −0
Large diff (134 lines) is not loaded by default to keep the page fast.Load diff
added packages/observers/src/dom/PageSummarizer.ts +50 −0
Large diff (50 lines) is not loaded by default to keep the page fast.Load diff
added packages/observers/src/dom/snapshot.page.js +143 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added packages/observers/src/index.ts +6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added packages/observers/src/network/NetworkObserver.ts +181 −0
Large diff (181 lines) is not loaded by default to keep the page fast.Load diff
added packages/observers/src/network/ResponseClassifier.ts +264 −0
Large diff (264 lines) is not loaded by default to keep the page fast.Load diff
added packages/observers/src/network/SchemaProfiler.test.ts +69 −0
Large diff (69 lines) is not loaded by default to keep the page fast.Load diff
added packages/observers/src/network/SchemaProfiler.ts +199 −0
Large diff (199 lines) is not loaded by default to keep the page fast.Load diff
added packages/platform-model/package.json +15 −0
Large diff (15 lines) is not loaded by default to keep the page fast.Load diff
added packages/platform-model/src/index.ts +254 −0
Large diff (254 lines) is not loaded by default to keep the page fast.Load diff
added packages/shared/package.json +12 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
added packages/shared/src/config.ts +56 −0
Large diff (56 lines) is not loaded by default to keep the page fast.Load diff
added packages/shared/src/index.ts +4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added packages/shared/src/logger.ts +37 −0
Large diff (37 lines) is not loaded by default to keep the page fast.Load diff
added packages/shared/src/types.ts +311 −0
Large diff (311 lines) is not loaded by default to keep the page fast.Load diff
added packages/shared/src/util.test.ts +34 −0
Large diff (34 lines) is not loaded by default to keep the page fast.Load diff
added packages/shared/src/util.ts +125 −0
Large diff (125 lines) is not loaded by default to keep the page fast.Load diff
added packages/storage/package.json +19 −0
Large diff (19 lines) is not loaded by default to keep the page fast.Load diff
added packages/storage/src/index.ts +0 −0

Binary file not shown.

added packages/storage/src/migrate.ts +12 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
added packages/storage/src/schema.sql +180 −0
Large diff (180 lines) is not loaded by default to keep the page fast.Load diff
added pnpm-lock.yaml +1381 −0
Large diff (1,381 lines) is not loaded by default to keep the page fast.Load diff
added pnpm-workspace.yaml +5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added tsconfig.json +34 −0
Large diff (34 lines) is not loaded by default to keep the page fast.Load diff
added vitest.config.ts +27 −0
Large diff (28 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere